msg_aa287ae79021 worker_done from structworker_26cac967-435a-4c40-b330-dd642cff8503
From mnema-talk v3.3.13, rendered on this site.
PR #777 WP2+WP7 review: no blocker, 2 should-fix, notes I reviewed the first four commits of PR #777 (22869571d, 2fb501c81, 226937115, 7fc2ba1c6) read-only at 7fc2ba1c6 in a scratch worktree and ran the requested tests. I found no blocker and no deadlock; there is one deterministic test failure in internal/safego with -count>1, one partly fixed earlier finding (S6 covers layered video only) and several notes. Nothing was edited, committed or pushed; what is left is the coordinator's decision on the should-fix items below.
TEST RUNS (at 7fc2ba1c6)
- go test -race -count=3 ./internal/ws/... : ok (45.8s), mediaframe ok. ./internal/sfu/... : ok (193s). No flake seen.
- go test -race -count=3 ./internal/safego/... : FAIL, deterministic, not a flake. Output, on run 2 and 3: "--- FAIL: TestRepeatedPanicsOfOneSiteAreCountedButNotAllLogged (0.00s) safego_test.go:204: logged 0 panics, want 5". Reproduced with -count=2 without -race; -count=1 passes.
- go test ./internal/repo/... : ok.
SHOULD-FIX
F1 (should-fix, test) internal/safego/safego_test.go:194-204 with safego.go:164 (package-global logs). The log budget of site "test_flood" is used up by the first run and lasts a minute, so every later run in the same process logs 0. Any -count>1 run fails (CI stress runs, the command in this task). Same latent problem in TestLoggedValueAndStack... ("test_bounds", fails from the 6th run). Fix: reset logs in the test (t.Cleanup restoring a fresh logBudget) or use a unique site name per run.
F2 (should-fix or document; S6 only partly fixed) internal/sfu/room_attach.go:227-243 (publishRemoteTrack -> forward with a webrtc.TrackLocalStaticRTP) versus layer_viewers.go:117-144 (writeTo/viewerWriteDone). The per-viewer recovery exists only on the layered path (layerGroup.write). Microphone audio, screen audio and a camera sent plain go through TrackLocalStaticRTP.WriteRTP, which runs every viewer's chain (the repo's own bweInterceptor, telemetryInterceptor, nackResponder history) on the publisher's forwarding goroutine. Scenario: a panic in one listener's bwe/telemetry writer while Alice speaks is recovered by "sfu_forward" with dropOnPanic(Alice): Alice is dropped for everybody, rejoins, and is dropped again while the faulty listener stays. Fix: wrap the per-binding writer the interceptors return (a first interceptor in the chain whose RTPWriter recovers and drops that viewer), or state the limit in the commit and in sfu_join.go:215.
S1-S7 AND THE LATER FIX S1 fixed. lifecycle_pumps.go:17-18 (two defers), lifecycle.go:322-327 (unregister defers forget), lifecycle.go:173-178 (abandon repeats close, cancelPending, forget, releaseSeat). Test TestPanicWhileUnregisteringStillRemovesTheClient is real: it forces a panic in startGrace and checks socket, online count, seat and room index. S2 fixed. lifecycle.go:147 runs serve under safego.Run("ws_handshake", c.abandon). Test TestPanicInTheGreetingClosesThatConnection covers a panic after registration. Note N1: h.noteUpgrade(true) (lifecycle.go:142) still runs after the hijack and outside safego; a panic in the UpgradeSeen hook leaves an open, unregistered socket (net/http does not close a hijacked connection). Move it into serve. S3 fixed. sfu_join.go:227-232 tells the owner unconditionally; notifyLost is once-only and kickLostPeer ignores a replaced peer (voice_kick.go:37). Test TestPanicAfterThePeerLeftTheRoomStillTellsItsOwner is adequate. S4 fixed. nack_responder.go:198-218: three panics, then gaveUp, the RTCP reader returns errNackGaveUp, feedbackEnded drops the peer. The test checks the counter, the reader and feedbackEnded separately; the link through Pion's sender.ReadRTCP returning that error unchanged is not exercised (I believe it holds, from memory of Pion v4). S5 fixed. valueLimit 512, stackLimit 4096, 5 lines per site and minute (safego.go:33-41, 96-101). Tests exist; see F1. Notes: fmt.Sprint builds the whole value before it is cut (safego.go:136); after the burst nothing says how many lines were suppressed (the counter still counts). S6 fixed for layered video only, see F2. Test TestPanickingViewerTrackDropsThatViewerNotThePublisher is strong (exactly one panic, publisher and other viewer keep flowing). TestLayeredWriteWithRecoveryDoesNotAllocate asserts 0 allocations through group.write and passed under -race; the deferred method call with a direct recover() is correct and the closure is only built after a panic. safego.Locked in viewerTrack.write (viewer_track.go:275) did not allocate either. S7 fixed. internal/repo/safego_test.go flags go statements, time/context AfterFunc also under an alias or as a value, any .Go selector outside safego, and dot imports of time/context/safego; TestBareStartsAreFound proves each. Notes: only internal/ and cmd/ are walked (api/api.go and loadtest/ are outside); runtime.SetFinalizer/AddCleanup are not covered; Pion and HTTP callbacks remain wrapped by hand. releaseSeat in abandon: present (voice_presence.go:136-150) and covered by the S1 test. No test for "the member's other connection keeps the seat" on this path.
DEADLOCKS: clean Order as coded: Client.voiceMu -> Hub.mu -> Client.mu; SFU.roomsMu -> Room.mu; Room.notifyMu is held while hub callbacks take Hub.mu.RLock. How I checked: read every caller of setVoice (voice_join.go:88 and :328, voice.go:139, voice_presence.go:25 and :144), forget, releaseSeat, abandon, every onPanic handler, the grace timer and all SFU-to-hub handlers; read every section that holds Hub.mu (takeSeat, forget, cancelPending, removePresence, setMuteState, whileSeated, presence.go, DisconnectUser/Session, Close, all RLock fan-outs).
- No caller of setVoice holds Hub.mu or a Client.mu. Under Hub.mu only Client.mu, gate/stage locks, deliver and shutdown are reached; no voiceMu, no SFU call, no second Hub.mu acquisition (no nested RLock that a waiting writer could block).
- No ws code calls the SFU while holding Hub.mu, so notifyMu -> Hub.mu.RLock cannot cycle.
- OnLost/notifyLost run the hub's callback on a goroutine of its own (room_tracks.go:360,371), so kickLostPeer takes voiceMu with no SFU lock held, while leaveVoiceLocked calls SFU.RemovePeer under voiceMu: no cycle.
- Every onPanic runs after the panicking frames unwound; the remaining plain Lock/Unlock pairs in ws and sfu have no call between them that can panic into a recover with the lock held (checked by listing every Lock not followed by defer). viewerWriteDone runs after vt.mu was released by safego.Locked.
- Test helpers: tests call setVoice outside Hub.mu (panic_isolation_test.go:191 then :193). Note N2: publishScoped and BroadcastChannel call the Tap hook under or next to Hub.mu.RLock (event_scope.go:54-70); a Tap that called back into the hub would deadlock behind a waiting writer, and setVoice is now one more writer. Not new; the hook is documented as "must not block".
ROOM INDEX: clean Client.voiceCh is written only in setVoice, under Hub.mu together with the index (client.go:236-249); forget removes the entry under the same lock (lifecycle.go:343-345); moveRoomLocked refuses unregistered clients (voice.go:147). So a client is never in two rooms and never listed after forget, and empty rooms are deleted. Paths checked: join, move, takeover (both evictSiblingConnections and endSiblingVoices), leave, kick, kickLostPeer, channel delete, permission revocation, DisconnectUser/DisconnectSession and Close (the index empties when the read loops unregister), forget-only after a failed clean-up, releaseSeat. TestRoomIndexAgreesWithAScanOfAllConnections (2000 random steps against the old scan) passed 3 times.
EVENTS OLD VERSUS NEW: no difference found
The old predicate (c in h.clients and currentVoice == channel) and h.rooms[channel] select the same set: unregistered connections were excluded before too, an evicted sibling tab has voice nil in both, and the sender's other tab still receives (only skip itself is left out, voice_mute.go:26-30). Only sendToVoiceRoomExcept and sendToVoiceUser changed; publishScoped with a voice scope still scans. A client given a voice room before registering would be missing from the index; production cannot do that (joins come from the read loop) and the tests were changed to setVoice after registering.
releaseSeat
- A member whose other connection is still in the Talk keeps the seat: removePresenceUnlessPresent with userInVoice (voice_presence.go:148).
- A grace leave is not cut short: seats with a pending grace are skipped (voice_presence.go:159).
- No double leave: removePresence reports only when the seat was there, decided under Hub.mu (voice_presence.go:47-74), also against a grace timer firing at the same time. N3 (note, narrow race) voice_presence.go:147-149. releaseSeat sweeps every seat of the user that has no grace and no seated connection. Another connection of the same user that is inside seatVoice between takeUserGraces/takeSeat and setVoice (voice_join.go:72-88) is in exactly that state. If connection A panics in that window, B's seat is removed and "leave" is broadcast; B then finishes its join, and for a resume (rejoin=true) no "join" is announced, so B has media but is listed nowhere until the next change. Needs two connections of one member and microsecond timing. Fix: set the connection's voice before or under the same lock as takeSeat, or have releaseSeat skip a user with a join in flight. N4 (note) after Hub.Close, startGrace is a no-op and h.voice still lists members, so a connection that panics during shutdown reports a leave to Presence (audit) while the database is closing (voice_presence.go:82 with :147). Skip releaseSeat when the hub is closed.
HALF-UPDATED STATE AFTER A RECOVERED PANIC N5 (note) forget (lifecycle.go:334-364): a panic after delete(h.clients, c) and before the online/active counters leaves the member online for good, because the repeat in abandon returns at "not known". Only shutdown and map operations lie between, so unlikely. N6 (note) attachPeer (voice_join.go:339-352): a panic after SFU.Join and before setPeer leaves a media peer the hub does not know; it goes only when its ICE fails. N7 (note) removePresence (voice_presence.go:65-73): a panic in the Presence hook or Tap after the seat was deleted loses the "leave" broadcast and the voice time of that stay; the others show the member until they refetch. N8 (note, design) per-member timers use safego.Crash: ws_voice_grace (voice_presence.go:89), ws_voice_time (voicegate.go:160,215,231), and perm_changed (perm/service.go:371) which runs the hub's kicks. The same leave code closes one connection when it runs on a read loop and ends the process when it runs on the grace timer. Any panic inside an onPanic (abandon -> releaseSeat -> RemovePeer/Presence/Tap) also ends the process, as documented. N9 (note) layer_viewers.go:140 with sfu_join.go:236: dropViewer drops the user's current peer by id; if the viewer rejoined in between, the new, healthy peer is dropped instead of the old one.
OUTSIDE THE DIFF (unchanged code, seen while tracing) X1 (note, medium confidence) voice_join.go:324: evictSiblingConnections evicts a sibling whose seat stamp is 0, although the comment above says a sibling still joining stays (endSiblingVoices at :129 checks != 0). Two tabs of one member joining the same Talk within a few milliseconds can evict each other: both end with no voice, the seat stays listed with no connection and no grace, and one media peer stays attached. Fix: add o.seatStamp() != 0 to the condition.
SCOPE OF READING: fully read safego.go, ws lifecycle.go, lifecycle_pumps.go, client.go, hub.go, voice.go, voice_presence.go, voice_join.go, voice_kick.go, voice_takeover.go, voice_mute.go, layers.go, media.go, the three ws test files of the diff, sfu sfu_join.go, nack_responder.go, viewer_track.go and the whole remaining sfu diff; perms.go, presence.go, stage.go, handshake.go, broadcast.go, event_scope.go in the parts that take locks; other ws/sfu files by lock and call-site search only. The safego call sites outside ws/sfu were listed, not reviewed.