Review: the flaky ws, sfu and macOS stage tests fixed at their cause (PR #816, head 26dc8c8b9)

From mnema-talk v3.3.13, rendered on this site.

Scope after the review. #816 was narrowed after this report: the fixture-wide change (section 3 and notes 2, 3 and 5) moved to a pull request of its own, which gets its own review. In #816, stage-switch.spec.ts now only adds trackPeerConnections to member B's context as well, which is the code gap section 1 names. The four Go test changes are unchanged; the commit hashes below are those of the reviewed head, before the branch was rebuilt.

Reviewer: an agent that did not write the change. Read-only; own detached worktree at 26dc8c8b9 (on current main), now removed; throwaway Docker PostgreSQL only. Commits reviewed: 1a1766769 (noise filter off in every e2e context, from the fixture), 037cfee4e (trickle and answer set test answers the real offer), 0aebad8dd (panic from the afterPeer hook), 26dc8c8b9 (best-effort pong in the lifecycle test). CODEOWNERS path touched: internal/sfu/ (a test file only). No product code changes. Mutations were made in the scratch worktree only and reverted after each run; nothing was committed or pushed.

Verdict

MERGE once the two CI jobs still pending at review time (e2e-shard (1/4), the ci.yml clip-sweep (2/4)) are green.

  • Each of the four Go fixes names the real cause. I reproduced two of them on main (one by forcing the timing) and saw the PR's version pass under the same conditions.
  • Each test still fails when the product behaviour it guards is broken.
  • None of the changes hides a product race. Each failure came from the test's own setup: a socket nobody reads, a made-up SDP, or a send queue the product never closes.
  • The fixture change removes no assertion from any spec. Every spec that reads window.__pcs gets it from contexts the fixture creates.

There are no blocking findings, only five notes (below).

1. Is the stated cause the real one?

TestWSWriterPeriodicLifecycle/revoked and /deleted (internal/ws/hub_database_integration_test.go:157-165)

  • The test's server-side connection has no read loop. The remote's pongs pile up unread, so when writePump closes the socket after the policy-violation close (lifecycle_pumps.go:81-86, :172-176), the kernel resets the connection.
  • If the remote reads a ping that came before the close frame and answers it, the pong write fails with "broken pipe" (or reset). On main, the ping handler returned that error. gorilla's advanceFrame passes a handler error straight back, so ReadMessage returned it and never reached the close frame.
  • Forced reproduction: I delayed the pong past the revocation (a 150 ms sleep in the handler after the third ping), 20 runs per variant.
    • With main's handler, the test failed with write tcp ...: write: broken pipe, want policy violation, which is the PR's error.
    • With the PR's handler it passed every time.
  • Without forcing, main did not fail on macOS in 600 subtest runs (-race -count=300, revoked and deleted). The PR reports 3 in 600 on loaded Linux.
  • Verdict: real cause. The fix is right: gorilla's default ping handler also treats the pong as best effort.

TestTrickleAndAnswerShareTheSet (internal/sfu/candidate_dedupe_test.go:85-115)

  • vetAnswerCandidates checks the answer's media sections against PC.PendingLocalDescription() (answer_candidates.go:35-39, :105-124). Join makes its offer on a goroutine of its own. When that offer was already pending, the old one-section sdpHead answer failed with errAnswerSections. When it was not yet pending, the check was skipped.
  • main, -race -count=200: 1 failure, vet: sfu: the answer's media sections are not the offer's. That is exactly the stated cause.
  • The PR waits for the offer, answers it with a real Pion client and appends the candidate line.
    • The answer carries no candidates of its own, because the client never calls SetLocalDescription and so never gathers. So len(seen) == 1 still measures only the trickled candidate and the answer's copy of it.
  • Product-wise there is nothing to hide here: a real client always answers the pending offer, and refusing a mismatched answer is the intended guard.

TestPanicAfterTheMediaJoinRemovesThePeer (internal/ws/voice_seat_cleanup_test.go:124-153)

  • On main, the test made the join panic by close(first.send).
    • Production never closes c.send (no close(c.send) outside tests; deliver checks closed and never panics), so this was a test device only.
    • The tab's own SFU offer and candidates are sent from separate goroutines (attachPeer sendOffer/sendICE). If one of them hits the closed queue first, its recovery takes the tab out of voice, and the second join then has no sibling to tell. That matches the reported "the join did not panic".
  • The new panic point is the existing afterPeer hook (voice_join.go:333-335), which runs right after seatPeer.
    • The old point was inside evictSiblingConnections. Both come after c.setPeer(peer) (voice_takeover.go:72), so the clean-up starts from the same state: the hub knows the peer and the SFU room holds it.
    • The new assertion c.peer() != nil pins that, so the hook cannot move before the attach unnoticed.
  • I did not reproduce main's failure (the PR also could not reproduce it). The mechanism follows from the code, and the new form has no goroutine left in the decision.

TestAbuseCapEndsTheMembersVoice

No change in the PR. Ran it -race -count=200: 0 failures.

stage-switch.spec.ts (macOS WebKit): only member A turned the noise filter off

On main, twoMembers added trackPeerConnections to aCtx only, so member B loaded DeepFilterNet3. The stated cause (20.5 s to B's camera track against 20 s waits) comes from CI run 43357593355, which I did not re-read. The code gap is real.

2. Does each test still fail when the product breaks?

TestMutation (scratch only, reverted)Result
Lifecycle revoked/deletedwriteSessionRevokedClose returns at once (no close frame)FAIL both: read: connection reset by peer, want policy violation
Lifecycle revokedrevocation check never true (tv != c.tokenVersion ignored)FAIL: i/o timeout, want policy violation
TrickleAndAnswerShareTheSettrickled candidates keyed apart from answer candidates (key = "t:" + key in admitRemoteCandidate)FAIL: distinct = 2, want 1
PanicAfterTheMediaJoinreleaseSeat no longer calls SFU.RemovePeer (voice_presence.go:197)FAIL: a media peer stayed in the Talk after its connection was cleaned up

The lifecycle test's heartbeats.Load() < 1 check is unchanged: the handler still counts every ping, even when the pong fails.

3. The fixture change (e2e/tests/fixtures.ts:240-241, helpers.ts:87-104)

  • Every context gets the recorder. counted wraps browser.newContext, including the webkit contextOwnProcess path. Every spec that reads window.__pcs (directly, or through expectTalkLive / audioDecodedIn) imports test from ./fixtures and creates its contexts with browser.newContext.
    • I compared the list of files that read __pcs with the files in the diff: all 26 such specs are in the diff and none is missing.
    • The one spec not on the fixture is page-errors.spec.ts, and it reads no __pcs.
    • No spec uses browser.newPage() or a browser launched by the spec itself.
  • Order of init scripts. The fixture adds the recorder when it creates the context, so it runs before every script the spec adds. I checked the four specs that set another filter (late-join, audio-stall, perf-idle set ai; opus-dtx sets browser): on main they already added their script after trackPeerConnections, so the effective setting is unchanged.
  • No assertion is lost.
    • Every removed line in the specs is an import of trackPeerConnections, an addInitScript(trackPeerConnections) call, a duplicate recorder body (hidden-video, screen-layers incl. keepPeers, talk-extras), or a comment.
    • The counts of expect(, test( and .poll( are the same as on main in all 26 specs.
  • Counts of __pcs.
    • talk-extras with sound: true had every connection in __pcs twice on main (two recorders). Its reads are find and flatMap(...).filter used for a maximum, so removing the duplicate does not change their meaning.
    • screen-layers' sealed.length >= 3 on the sharer had one recorder before and has one now.
    • The length checks in voice-takeover, key-lifecycle, late-join and browser-sealing compare before and after (or with 0) on contexts that had exactly one recorder before.
  • Gates.
    • npm run typecheck (e2e): pass.
    • ESLint with web/eslint.config.ts on e2e: pass.
    • CI on 26dc8c8b9: 40 checks pass, including all six webkit-macos-calls groups, firefox, webkit 1/2 and 2/2, and backend-integration. Two were pending at review time (named in the Verdict).

4. Runs

WhatCommandResult
PR, ws unitgo test -race -count=200 -run 'TestAbuseCapEndsTheMembersVoice|TestPanicAfterTheMediaJoinRemovesThePeer' ./internal/ws/ok
PR, ws integrationgo test -tags=integration -race -count=200 -run TestWSWriterPeriodicLifecycle ./internal/ws/ok (272 s, 800 subtests)
PR, sfugo test -race -count=200 -run 'TestTrickleAndAnswerShareTheSet$' ./internal/sfu/ok
main, sfu testsame, main's test file1 of 200 failed (the stated error)
main, lifecycle-race -count=300, revoked and deleted0 of 600 failed (not reproduced unforced)
Forced late pong20 runs per variantmain's handler fails ("broken pipe"), PR's passes

All runs used nice -n 10, waiting while the load was above 12. Note: TestWSWriterPeriodicLifecycle is in an integration-tagged file, so a plain go test -run skips it silently.

Notes (none blocking)

  1. Lifecycle: the reset is the root, and the PR now tolerates it. The pong fails because the test's server conn is never read, so the close turns into a reset. In production readPump drains the socket, so there is no product race. In principle a reset could also make the remote's read fail before it takes the close frame from the kernel buffer. I did not see that in 400 revoked/deleted runs on the PR, and the PR's 1000 runs did not show it either. If it ever shows up, a drain goroutine on server in the test (as a real read loop would be) removes the reset itself.
  2. mnema_noise=off is now written at every page load of every context, in non-call specs too (helpers.ts:103). helpers.ts says a spec that checks the filter turns it back on with its own init script. That covers checking the default, but not a future spec that picks a filter in the settings dialog and reloads to see that it persisted: the init script resets it on each load. Today no spec touches the noise setting (grep for noise, the German labels and the setting's test ids in e2e/tests). One sentence in the comment would prevent the surprise.
  3. The AI filter now runs in fewer specs. Contexts that had no recorder on main ran DeepFilterNet3 by accident: stage-switch member B, hidden-video, screen-layers. They no longer do. The filter path itself stays covered, CSP included through the auto fixture, by late-join (which checks the deepfilter-worker-bridge), perf-idle and audio-stall. This is intended, but worth knowing.
  4. The panic test no longer goes through a sibling tab. It now proves clean-up after a panic at the hook, not at "telling the other tab". Both points come after setPeer, so for the clean-up they are the same. The eviction path itself is covered by the takeover tests.
  5. The recorder wraps RTCPeerConnection in every frame of every context, also in specs that never call. This is harmless (a subclass, no behaviour change), and the new try around localStorage covers frames without storage.

Not checked

  • I did not run e2e or call specs locally (rule: CI only, no camera, microphone or audio). The macOS stage-switch claim rests on the code gap and on the PR's CI being green.
  • I did not re-read the CI logs of runs 43293325602 and 43357593355 that the PR cites.
  • I did not run the PR's load setup (-cpu 1,2, CPU burners) on Linux.

Outcome

Merge after the pending checks pass. The notes need no change before merge. Note 2 is a one-line comment, if the author wants it.

Something wrong on this page? Tell us.