Review: the flaky ws, sfu and macOS stage tests fixed at their cause (PR #816, head 26dc8c8b9)
From mnema-talk v3.3.13, rendered on this site.
Scope after the review. #816 was narrowed after this report: the fixture-wide change (section 3 and notes 2, 3 and 5) moved to a pull request of its own, which gets its own review. In #816, stage-switch.spec.ts now only adds trackPeerConnections to member B's context as well, which is the code gap section 1 names. The four Go test changes are unchanged; the commit hashes below are those of the reviewed head, before the branch was rebuilt.
Reviewer: an agent that did not write the change. Read-only; own detached worktree at 26dc8c8b9 (on current main), now removed; throwaway Docker PostgreSQL only.
Commits reviewed: 1a1766769 (noise filter off in every e2e context, from the fixture), 037cfee4e (trickle and answer set test answers the real offer), 0aebad8dd (panic from the afterPeer hook), 26dc8c8b9 (best-effort pong in the lifecycle test).
CODEOWNERS path touched: internal/sfu/ (a test file only). No product code changes.
Mutations were made in the scratch worktree only and reverted after each run; nothing was committed or pushed.
Verdict
MERGE once the two CI jobs still pending at review time (e2e-shard (1/4), the ci.yml clip-sweep (2/4)) are green.
- Each of the four Go fixes names the real cause. I reproduced two of them on main (one by forcing the timing) and saw the PR's version pass under the same conditions.
- Each test still fails when the product behaviour it guards is broken.
- None of the changes hides a product race. Each failure came from the test's own setup: a socket nobody reads, a made-up SDP, or a send queue the product never closes.
- The fixture change removes no assertion from any spec. Every spec that reads
window.__pcsgets it from contexts the fixture creates.
There are no blocking findings, only five notes (below).
1. Is the stated cause the real one?
TestWSWriterPeriodicLifecycle/revoked and /deleted (internal/ws/hub_database_integration_test.go:157-165)
- The test's server-side connection has no read loop. The remote's pongs pile up unread, so when
writePumpcloses the socket after the policy-violation close (lifecycle_pumps.go:81-86, :172-176), the kernel resets the connection. - If the remote reads a ping that came before the close frame and answers it, the pong write fails with "broken pipe" (or reset). On main, the ping handler returned that error. gorilla's
advanceFramepasses a handler error straight back, soReadMessagereturned it and never reached the close frame. - Forced reproduction: I delayed the pong past the revocation (a 150 ms sleep in the handler after the third ping), 20 runs per variant.
- With main's handler, the test failed with
write tcp ...: write: broken pipe, want policy violation, which is the PR's error. - With the PR's handler it passed every time.
- With main's handler, the test failed with
- Without forcing, main did not fail on macOS in 600 subtest runs (
-race -count=300, revoked and deleted). The PR reports 3 in 600 on loaded Linux. - Verdict: real cause. The fix is right: gorilla's default ping handler also treats the pong as best effort.
TestTrickleAndAnswerShareTheSet (internal/sfu/candidate_dedupe_test.go:85-115)
vetAnswerCandidateschecks the answer's media sections againstPC.PendingLocalDescription()(answer_candidates.go:35-39, :105-124).Joinmakes its offer on a goroutine of its own. When that offer was already pending, the old one-sectionsdpHeadanswer failed witherrAnswerSections. When it was not yet pending, the check was skipped.- main,
-race -count=200: 1 failure,vet: sfu: the answer's media sections are not the offer's. That is exactly the stated cause. - The PR waits for the offer, answers it with a real Pion client and appends the candidate line.
- The answer carries no candidates of its own, because the client never calls SetLocalDescription and so never gathers. So
len(seen) == 1still measures only the trickled candidate and the answer's copy of it.
- The answer carries no candidates of its own, because the client never calls SetLocalDescription and so never gathers. So
- Product-wise there is nothing to hide here: a real client always answers the pending offer, and refusing a mismatched answer is the intended guard.
TestPanicAfterTheMediaJoinRemovesThePeer (internal/ws/voice_seat_cleanup_test.go:124-153)
- On main, the test made the join panic by
close(first.send).- Production never closes
c.send(noclose(c.send)outside tests;deliverchecksclosedand never panics), so this was a test device only. - The tab's own SFU offer and candidates are sent from separate goroutines (attachPeer sendOffer/sendICE). If one of them hits the closed queue first, its recovery takes the tab out of voice, and the second join then has no sibling to tell. That matches the reported "the join did not panic".
- Production never closes
- The new panic point is the existing
afterPeerhook (voice_join.go:333-335), which runs right afterseatPeer.- The old point was inside
evictSiblingConnections. Both come afterc.setPeer(peer)(voice_takeover.go:72), so the clean-up starts from the same state: the hub knows the peer and the SFU room holds it. - The new assertion
c.peer() != nilpins that, so the hook cannot move before the attach unnoticed.
- The old point was inside
- I did not reproduce main's failure (the PR also could not reproduce it). The mechanism follows from the code, and the new form has no goroutine left in the decision.
TestAbuseCapEndsTheMembersVoice
No change in the PR. Ran it -race -count=200: 0 failures.
stage-switch.spec.ts (macOS WebKit): only member A turned the noise filter off
On main, twoMembers added trackPeerConnections to aCtx only, so member B loaded DeepFilterNet3. The stated cause (20.5 s to B's camera track against 20 s waits) comes from CI run 43357593355, which I did not re-read. The code gap is real.
2. Does each test still fail when the product breaks?
| Test | Mutation (scratch only, reverted) | Result |
|---|---|---|
| Lifecycle revoked/deleted | writeSessionRevokedClose returns at once (no close frame) | FAIL both: read: connection reset by peer, want policy violation |
| Lifecycle revoked | revocation check never true (tv != c.tokenVersion ignored) | FAIL: i/o timeout, want policy violation |
| TrickleAndAnswerShareTheSet | trickled candidates keyed apart from answer candidates (key = "t:" + key in admitRemoteCandidate) | FAIL: distinct = 2, want 1 |
| PanicAfterTheMediaJoin | releaseSeat no longer calls SFU.RemovePeer (voice_presence.go:197) | FAIL: a media peer stayed in the Talk after its connection was cleaned up |
The lifecycle test's heartbeats.Load() < 1 check is unchanged: the handler still counts every ping, even when the pong fails.
3. The fixture change (e2e/tests/fixtures.ts:240-241, helpers.ts:87-104)
- Every context gets the recorder.
countedwrapsbrowser.newContext, including the webkitcontextOwnProcesspath. Every spec that readswindow.__pcs(directly, or throughexpectTalkLive/audioDecodedIn) importstestfrom./fixturesand creates its contexts withbrowser.newContext.- I compared the list of files that read
__pcswith the files in the diff: all 26 such specs are in the diff and none is missing. - The one spec not on the fixture is
page-errors.spec.ts, and it reads no__pcs. - No spec uses
browser.newPage()or a browser launched by the spec itself.
- I compared the list of files that read
- Order of init scripts. The fixture adds the recorder when it creates the context, so it runs before every script the spec adds. I checked the four specs that set another filter (late-join, audio-stall, perf-idle set
ai; opus-dtx setsbrowser): on main they already added their script aftertrackPeerConnections, so the effective setting is unchanged. - No assertion is lost.
- Every removed line in the specs is an import of
trackPeerConnections, anaddInitScript(trackPeerConnections)call, a duplicate recorder body (hidden-video, screen-layers incl.keepPeers, talk-extras), or a comment. - The counts of
expect(,test(and.poll(are the same as on main in all 26 specs.
- Every removed line in the specs is an import of
- Counts of
__pcs.- talk-extras with
sound: truehad every connection in__pcstwice on main (two recorders). Its reads arefindandflatMap(...).filterused for a maximum, so removing the duplicate does not change their meaning. - screen-layers'
sealed.length >= 3on the sharer had one recorder before and has one now. - The
lengthchecks in voice-takeover, key-lifecycle, late-join and browser-sealing compare before and after (or with 0) on contexts that had exactly one recorder before.
- talk-extras with
- Gates.
npm run typecheck(e2e): pass.- ESLint with
web/eslint.config.tsone2e: pass. - CI on 26dc8c8b9: 40 checks pass, including all six
webkit-macos-callsgroups, firefox, webkit 1/2 and 2/2, and backend-integration. Two were pending at review time (named in the Verdict).
4. Runs
| What | Command | Result |
|---|---|---|
| PR, ws unit | go test -race -count=200 -run 'TestAbuseCapEndsTheMembersVoice|TestPanicAfterTheMediaJoinRemovesThePeer' ./internal/ws/ | ok |
| PR, ws integration | go test -tags=integration -race -count=200 -run TestWSWriterPeriodicLifecycle ./internal/ws/ | ok (272 s, 800 subtests) |
| PR, sfu | go test -race -count=200 -run 'TestTrickleAndAnswerShareTheSet$' ./internal/sfu/ | ok |
| main, sfu test | same, main's test file | 1 of 200 failed (the stated error) |
| main, lifecycle | -race -count=300, revoked and deleted | 0 of 600 failed (not reproduced unforced) |
| Forced late pong | 20 runs per variant | main's handler fails ("broken pipe"), PR's passes |
All runs used nice -n 10, waiting while the load was above 12. Note: TestWSWriterPeriodicLifecycle is in an integration-tagged file, so a plain go test -run skips it silently.
Notes (none blocking)
- Lifecycle: the reset is the root, and the PR now tolerates it. The pong fails because the test's server conn is never read, so the close turns into a reset. In production
readPumpdrains the socket, so there is no product race. In principle a reset could also make the remote's read fail before it takes the close frame from the kernel buffer. I did not see that in 400 revoked/deleted runs on the PR, and the PR's 1000 runs did not show it either. If it ever shows up, a drain goroutine onserverin the test (as a real read loop would be) removes the reset itself. mnema_noise=offis now written at every page load of every context, in non-call specs too (helpers.ts:103). helpers.ts says a spec that checks the filter turns it back on with its own init script. That covers checking the default, but not a future spec that picks a filter in the settings dialog and reloads to see that it persisted: the init script resets it on each load. Today no spec touches the noise setting (grep fornoise, the German labels and the setting's test ids ine2e/tests). One sentence in the comment would prevent the surprise.- The AI filter now runs in fewer specs. Contexts that had no recorder on main ran DeepFilterNet3 by accident: stage-switch member B, hidden-video, screen-layers. They no longer do. The filter path itself stays covered, CSP included through the auto fixture, by late-join (which checks the
deepfilter-worker-bridge), perf-idle and audio-stall. This is intended, but worth knowing. - The panic test no longer goes through a sibling tab. It now proves clean-up after a panic at the hook, not at "telling the other tab". Both points come after
setPeer, so for the clean-up they are the same. The eviction path itself is covered by the takeover tests. - The recorder wraps
RTCPeerConnectionin every frame of every context, also in specs that never call. This is harmless (a subclass, no behaviour change), and the newtryaroundlocalStoragecovers frames without storage.
Not checked
- I did not run e2e or call specs locally (rule: CI only, no camera, microphone or audio). The macOS stage-switch claim rests on the code gap and on the PR's CI being green.
- I did not re-read the CI logs of runs 43293325602 and 43357593355 that the PR cites.
- I did not run the PR's load setup (
-cpu 1,2, CPU burners) on Linux.
Outcome
Merge after the pending checks pass. The notes need no change before merge. Note 2 is a one-line comment, if the author wants it.