Review of PR #780 (msch128/audit-train-3), first six commits

From mnema-talk v3.3.13, rendered on this site.

Read-only review at 38ff74a2f (commit 6) against origin/main 860cce535. Nothing edited, committed or pushed. Commits: f6e57f2b1 (SFU viewer guard), 48a5f4254 (Talk seat), d6ab3c2ca (key rotation), 01645f0eb (read marker), abddf0960 (bot slow mode), 38ff74a2f (bot upload mentions). The picture-metadata commit was not reviewed.

VERDICT: no blocker. 1 should-fix (R1, a race that the seat commit narrows but does not close), 7 notes. Every finding of the two earlier reviews that was asked about is fixed, except N3 of the ws review, which is partly fixed (R1).

Test runs (scratch worktree at 38ff74a2f, removed afterwards)

  • go build ./...: ok. go vet ./...: ok. go vet -tags=integration ./...: ok. gofmt on the changed files: clean.
  • go test -race -count=2 ./internal/ws/... ./internal/sfu/... ./internal/safego/...: ok (ws 31 s, sfu 129 s, safego 2.4 s). safego failed here with -count=2 before (F1).
  • go test ./internal/auth/... ./internal/config/... ./cmd/...: ok.
  • go test -tags=integration ./internal/auth/ ./cmd/server/ ./internal/config/ (Postgres via Docker): ok.
  • go test -tags=integration ./internal/server/ filtered to slow mode, read marker, polls, slash commands, mentions, bot upload and key rotation: 62 tests, all pass, among them the new ones (TestSlowModeHoldsABotBackUnlessItMayManageMessages, TestSlowModeAppliesToACommandResponseLikeToTheBotsMessages, TestABuiltInTextCommandMarksTheChannelReadLikeATypedMessage, TestPostingAPollMarksTheChannelRead, TestAScheduledMessageLeavesWhatTheAuthorHasNotReadUnread, TestBotUploadCaptionMentionsFollowAllowedMentions, TestServerAcceptsSessionsSignedWithThePreviousSecret, TestServerSealsAuthenticatorSecretsUnderTheDataKey).
  • Guard benchmark (not under -race): guarded 4.5 ns/op, bare 1.5 ns/op, 0 allocs/op in both.
  • Three stress tests of my own in internal/ws (scratch only, not kept), run on the PR and on origin/main:
Stress testorigin/mainPR
two tabs join the same Talk at once, SFU on: "neither tab seated"392 of 4000 of 30,000
same: "both seated" / "winner holds a replaced media peer"0 / 00 / 0
tab A panics (abandon) while tab B joins: B holds voice, member listed nowhere200 of 200,00071 of 200,000
tab A leaves normally while tab B joins: same end state1,327 of 200,000962 of 200,000

Findings

R1 (should-fix; N3 of the ws review is narrowed, not closed) internal/ws/voice_presence.go:35-39 (removePresenceUnlessPresent) with :156-158 (releaseSeat) and voice.go:140 (leaveVoiceLocked). The check "is another connection of the member in the Talk" (userInVoice, read lock) and the removal (removePresence, write lock) are two separate lock sections. takeSeat now sets seat and connection under one lock, which closes the window N3 named, but a join that takes its seat between the check and the removal still loses it. Scenario: tab A is seated. A panics (or simply leaves) while tab B joins. A: setVoice(nil), userInVoice = false. B: takeSeat finds the seat still listed, so rejoin = true, and sets its voice. A: removePresence deletes the seat and broadcasts "leave". B finishes silently (a rejoin is not announced): B has media and believes it is in the Talk, the others do not see the member, the user limit does not count them, the stay's voice time is lost, and B's own later leave is not announced. Reproduced: 71 of 200,000 on the panic path, 962 of 200,000 on the ordinary leave path (table above; both exist on main, so nothing is new, but the commit message says the seat can no longer be taken away). Fix: decide inside removePresence's h.mu section. h.rooms[chID] and Client.voiceCh are both kept under h.mu now, so "remove unless a registered connection of this user other than except is in h.rooms[chID]" is one loop under the write lock. The same check-then-act is in dropVoiceForGrace (voice_presence.go:28-30, unchanged code, read only, not reproduced): userInVoice = false, the reconnect takes its seat, then startGrace arms a timer whose removePresence (:100-101) does not look for a seated connection, so a resumed member would drop out of the list after the grace time. Starting the grace under the same lock as the check fixes that too.

R2 (note) internal/ws/voice_join.go:341-354. The winner of two simultaneous joins is decided by seatClock, the media peer by the order of SFU.Join; nothing ties the two. Interleaving: A's SFU.Join, B's SFU.Join (replaces A's peer), B stamps 1 and evicts nobody (A has stamp 0, the X1 rule), A stamps 2 and evicts B. Result: A keeps the seat with a peer the SFU has already replaced and closed, B is told joined_elsewhere while its peer is the room's live one until its ICE times out (kickLostPeer then ignores it). Never both seated, never two peers; the member is listed but silent until tab A's client rejoins. Not observed in 30,000 concurrent joins; on main the same interleaving ended with both tabs evicted. Fix, if wanted: order by the peer's own number (sfu.Peer.gen is assigned under Room.mu in attachPeer, so it is the replacement order) instead of a second clock.

R3 (note) internal/sfu/sfu_join.go:317-333 with Pion peerconnection.go:122. The guard finds its peer by PeerConnection.ID(), which is "PeerConnection-" + time.Now().UnixNano(). Connections are built concurrently before any lock (sfu_join.go:49-50), so two can share an ID (same nanosecond; more likely on a clock with coarser steps). Then dropConnection keeps the last match: the healthy peer may be dropped while the panicking one stays with a dead guard, receiving nothing and never told. Fix: drop every match, or give the guard its peer directly (attachPeer registers the peer under the ID in a map the factory reads).

R4 (note) internal/auth/key_rotation.go:133-141. rotate-keys deletes every unconfirmed enrolment its own key set cannot read. When it runs with other settings than the server (the S1 mistake) and every confirmed secret is still readable, a setup a member started minutes ago under the server's key is deleted; the member's confirmation then fails and they start again. Low impact, and the output names the account. A confirmed row cannot be taken for an unconfirmed one: the rows are locked FOR UPDATE, confirmed_at IS NOT NULL is read under that lock, and the DELETE repeats confirmed_at IS NULL. Fix, if wanted: only delete setups older than the setup's own lifetime (for example an hour).

R5 (note) cmd/server/main.go:136 with internal/auth/key_check.go:76-94. The start-up check has no time limit of its own (doctor has doctorProbeTimeout) and runs before the server listens. Cost: one sequential read of user_totp (confirmed rows), streamed row by row, one AES-GCM open per configured key per row, so about a second per several hundred thousand accounts with two-factor sign-in; fine for this product. A bounded context would keep a slow database from delaying the start.

R6 (note) internal/auth/key_check.go:18-20. The fingerprint is 32 bits of HMAC(root, fixed text) and now goes into the server log, doctor and rotate-keys output. For an instance without DATA_KEY the root is JWT_SECRET. It is a check value for a guessed secret, but any session token already is one, and production enforces 32 characters; no real advantage. No key material and no member name is logged at start (count only); rotate-keys names accounts as it did before, now also the removed setups.

R7 (note) internal/media/botapi.go:68-77 and stream.go:29. (a) allowed_mentions=null is taken as absent (strict default: members only), which is safe. (b) A member's upload now accepts and ignores an allowed_mentions field; harmless, and api/openapi.json documents the field only for the bot route.

R8 (note) internal/sfu/sfu_join.go:279-281. Only the RTP writer is guarded. RTCP readers run on the viewer's own goroutine (sfu_rtcp, dropOnPanic(viewer)) and retransmissions of the NACK responder are written beneath the guard on that same goroutine, so both already stay with the viewer. Goroutines Pion's interceptors start themselves (sender reports, transport-cc feedback, NACK generator) are outside any recovery, as before.

Commit 1, internal/sfu: verified

  • F2 fixed. buildInterceptors adds viewerGuardFactory last (sfu_setup.go:219). Pion's Chain.BindLocalStream (interceptor v0.1.51 chain.go:38-44) wraps in registration order, so the last one added is the first writer a packet passes; RTPSender binds the chain around the SRTP write (webrtc v4.2.26 rtpsender.go:362-369), so SRTP, the bandwidth estimate, the transport-cc stamp, telemetry and the NACK history are all beneath it. TestEveryConnectionsChainStartsWithTheViewerGuard checks this on the real registry.
  • Correct viewer, not the publisher: the guard returns io.ErrClosedPipe after a panic; TrackLocalStaticRTP collects per-binding errors and goes on to the next binding; forward ignores ErrClosedPipe (room_attach.go:338-343). The drop runs on its own goroutine and goes by connection, so a rejoined viewer's new connection has its own guard and stays (see R3 for the one exception).
  • N9 fixed: dropViewerPeer compares the peer's gen (sfu_join.go:246), the viewer track carries it (room_signal.go:306). Test TestPanickingTrackOfAReplacedConnectionLeavesTheNewOne.
  • Per-packet path: one atomic load, one open-coded defer, no lock, 0 allocations (test and benchmark above).
  • Interceptor order and what they see: the guard passes header, payload and attributes through untouched, ignores the StreamInfo (each interceptor still gets the same pointer from the chain) and embeds NoOp for RTCP readers and writers and the remote streams. Nothing beneath it sees a difference.
  • Locks: dropConnection takes SFU.roomsMu then Room.mu (read), the documented order, and only on its own goroutine after a panic.

Commit 2, internal/ws and safego: verified

  • F1 fixed (refillLogBudget in captureLog; -count=2 passes). N1 fixed (noteUpgrade inside serve, under ws_handshake; test present). N4 fixed (releaseSeat returns when the hub is closed, after the media peer is removed; test). N5 fixed (counters directly after the delete). N6 fixed (setPeer before the stamp and the eviction; test). N7 fixed (leave broadcast and voice time deferred; same order as before; test). X1 fixed (stamp != 0 in evictSiblingConnections; table above). N3: see R1.
  • Lock order is still Client.voiceMu -> Hub.mu -> Client.mu. setVoiceLocked has exactly two callers: setVoice (takes Hub.mu itself) and takeSeat (holds Hub.mu.Lock). Under the lock takeSeat reaches only the *Locked helpers, Client.mu and the unchanged SendEvent; no path takes Hub.mu twice. releaseSeat asks isClosed after it released voiceMu and outside Hub.mu.
  • Interleavings of two connections A and B of one member in Talk T:
    1. A seated, B joins: B's SFU join replaces A's peer, B stamps, A is evicted and told. One seat, one peer.
    2. A and B join at once, stamps in SFU order: the later one wins, the earlier is told (test, stress 0/30,000).
    3. A and B join at once, stamps against SFU order: R2.
    4. A seated and renegotiating (stamp back to 0) while B joins: B does not evict A, A finishes later and evicts B. Later wins.
    5. A panics while B resumes, before B's takeSeat: seat removed, "leave" sent, B's takeSeat finds no seat and is announced as a join (test).
    6. A panics after B's takeSeat and before A's check: B stands for the seat, it stays, no "leave" (test).
    7. A panics or leaves, B's takeSeat falls between A's check and A's removal: R1.
    8. A panics between its SFU join and its eviction of B (nothing in between can panic today): A's peer is removed by abandon, B keeps the seat with a replaced peer until its client rejoins.
    9. A and B panic at once: both clean-ups call removePresence, which decides "was listed" under Hub.mu, so exactly one "leave".
    10. A panics after Hub.Close: peer removed, no leave reported (test).
    11. A's socket drops (grace) while B reconnects: the grace variant of R1, by reading only.
  • X1 cannot leave two tabs with a seat or a peer each when the SFU is on: every finished join stamps and evicts every sibling with a lower non-zero stamp, a sibling with stamp 0 evicts on its own finish, and the SFU keeps one peer per member and Talk. Without an SFU two tabs can both be seated, as before.

Commit 3, key rotation: verified

  • S1 fixed: rotate-keys prints the sealing key with a fingerprint and the previous keys, warns when none is configured and nothing moved; the server logs the same fingerprint, doctor prints it; the leak runbook is numbered with the restart before rotate-keys.
  • S2 fixed, including the way back. With DATA_KEY_PREVIOUS alone, sealingKey is the session-derived key (two_factor_keys.go:51-56): the previous data key is only in readKeys and never seals or hashes a new recovery code. A blob that names a key id opens only with a configured key of that id under AES-GCM (header and account id are associated data), and otherwise falls to the id-0 keys with the whole blob, which fails: an unknown-key blob cannot open. Validation is not weaker: DATA_KEY_PREVIOUS alone must still differ from both session secrets and meet the 32-character rule in production and staging, and FromEnv still refuses the start (the checks moved into RequiredProblems, which FromEnv calls; values are trimmed the same way).
  • N1 fixed (R4 for the one remark). N2 fixed: new codes store the fingerprint, the count leaves out codes whose fingerprint no configured key has, codes without one stay counted. N3 fixed: the step is spent first, the re-seal is best effort; both callers of VerifyFactor pass the pool, not a transaction, so a failed re-seal cannot abort the step update. N6 fixed (setup pre-check). N7 fixed (both sentences are in the leak section).
  • Migration 0102: one nullable BIGINT column without a default (metadata only, no table rewrite, a brief exclusive lock), additive, an older binary still inserts without it, forward-only like every migration here.

Commits 4-6, posting entry points: verified

  • Read marker: slashcmd.post and poll.Create call MarkReadAfterSend with the stored message, so the member is the author; MarkReadTo writes with GREATEST, so the marker never moves back; thread replies are skipped; the scheduled send does not call it (comment and test).
  • Slow mode: the exemption is p.Has(perm.ManageMessages) from the bot's own channel permissions (botapi.go:187-189), on the message route, the command response and the upload (gatedContext). Without the permission the flag is false and dm_gate.go:44 applies slow mode (test: 429 for the plain bot). The same flag lets a bot with Manage messages reply under a locked forum post, as for a member; doc/bots.md says so.
  • Bot upload caption: BotMentionPolicy is the old botMentionPolicy moved unchanged; @all and @here need both parse "everyone" and Mention everyone, roles come from MentionableRoles narrowed by the request, so the request only narrows. The permission check (Send messages and Attach files), slow mode and the new-member wait are the same as for a member's upload. A malformed field answers 400 and the stored file is discarded.
  • api/openapi.json: 4 added lines (one optional form property), no removed line.

Something wrong on this page? Tell us.