Phase 2 and fix review: H-L8-09 (subscription switches keep memory per switch)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: independent (I wrote neither the finding nor the fix). Finding: findings/H-L8-09.md (S3 claimed). Fix branch: origin/msch128/fix-h-l8-09, on origin/msch128/fix-h-l8-07 at e365c714d:

  • 1c787fdc2: test, a known failure in internal/sfu/subscription_cycle_test.go.
  • ff88d92aa: merge of msch128/fix-h-l8-03, the already reviewed fix that forgets the bandwidth estimator's writer of a stopped sender.
  • 714225dd4: fix, a media connection is closed past 32 768 senders made on it; the test now asserts a bound of 1.5 KB per switch.

All runs were on loopback, in temporary worktrees that are now removed, with nice -n 10 and -p 1, one heavy job at a time, at a load under 12. No audio device, camera or microphone was used.

Phase 2 verdict: STANDS (S3), with corrected figures. Not a duplicate. Fix verdict: MERGE AFTER FIXES (two blockers).


Phase 2

Reproduction

On 1c787fdc2, go test ./internal/sfu/ -run TestSubscriptionSwitches -v measured +3 175 bytes per switch over 1 000 switches. Goroutines +0, offer 26 984 → 26 984 bytes. The inverse assertion holds.

Is it documented?

No. Before the fix, SECURITY.md (media connection limits) said nothing about per-sender state kept for the life of a connection.

Is it a duplicate of H-L8-03 or H-L8-05?

H-L8-03. At ff88d92aa (H-L8-03's fix merged, no cap yet), the same test at 4 000 switches measures +265 bytes per switch, down from 3 175. So about 92% of the measured retention is H-L8-03's defect: the stopped sender's writer chain, pinned by the pacer. H-L8-09's headline figure is H-L8-03 seen from the member's side.

H-L8-09 is still a finding of its own, for two reasons:

  • Reach. H-L8-03 is rated S4: passive churn, "no attacker needed". H-L8-09 is an ordinary member driving the same retention on purpose, at the speed of its own events.
  • What is left. H-L8-03's fix does not touch the residual of a few hundred bytes per sender (Pion's per-SSRC state and the rest). Without the cap it is unbounded for the life of the connection and drivable by the member.

H-L8-05 is a different path: state kept on a viewer's connection per member who joined and left. Its known failure is unchanged on the fix tip: 25 470 bytes per joiner at 1 200 rounds, inverse assertion still holds.

Severity

S3 stands. Ordinary members can bypass the per-connection bounds SECURITY.md states, in both states:

  • Before any fix: with camera batches of 16 (SubscribeCameras, the path of the batch event), I measured 6 843 bytes per sender. At the WebSocket budget of 40 events a second (20 off/on toggles, 320 senders a second) that is about 2.2 MB a second per member.
  • With H-L8-03's fix only: about 273 bytes per sender, both ends of the connection (measured on the tip below, which differs from the H-L8-03-only state by the cap alone). At the event budget that is about 87 KB a second per member, unbounded, about 1 GB in 3.4 hours, times the number of members.

Corrections to the claim:

  • "16.4 KB per event" includes warm-up. The steady figure is about 3.2 KB per switch on a screen share, about 6.8 KB per sender.
  • "About 10 MB a second" with camera batches is about 4.5 times too high. Measured and scaled to the event budget it is about 2.2 MB a second.

Fix review (tip 714225dd4)

What still grows, measured

I diffed heap profiles taken after two forced GCs (MemProfileRate 1024) and fitted the heap in use per switch.

  • Screen share (the fix's own test, 2 000 switches): +118 to +430 bytes per switch, depending on run and load (118 under -race). Goroutines +0, offer flat.
  • Cameras, not measured by the author:
    • 4 publishers, 600 toggles, 2 400 senders: +836 bytes per sender, with the curve flattening.
    • 16 publishers (full batches), 300 toggles, 4 800 senders: +273 bytes per sender for both ends. That matches the author's 160 to 240 bytes on the server side.
  • The profile diff on the server side shows no stopped sender and no writer chain. What remains is SDP and codec parse objects of the current descriptions (live in both snapshots, noise) and a small SRTP/SRTCP share.
  • In rapid toggles most senders never encrypt a packet before they are stopped, so the SRTP state per sender is below the author's 156 bytes. Either way, the cost is a few hundred bytes per sender, and the cap bounds it.
  • Harness note: in my first camera runs the publishers' own NACK send history filled up during the run (rtpbuffer copies under the test client's publish loop). That is test-client growth, not the SFU. Filling both tracks' histories first removes it. The fix's screen-share test already does this (fillSendHistory).

Pace

Measured on the tip with batches of 16 (in process, no event budget) on this machine, at a load of about 10: about 164 senders a second, limited by the renegotiation round trip. Before the fix the same probe on a quieter machine ran at about 970 a second. The author's figure, 320 senders a second at the event budget, is the right upper bound to plan with. At that pace one member reaches the cap in about 100 to 200 seconds.

BLOCKER B1: the cap closes the connection silently

trip calls PC.Close(). peerStateChanged treats the closed state with removePeer alone; only the failed state goes through dropLostPeer → notifyLost. I checked it with a probe: after the sender cap tripped, the peer's OnLost callback did not fire within 15 s, and the test client's connection went to closed.

So the hub is never told:

  • The member keeps the call seat, and the others see them in the call.
  • The member gets no voice_kicked and no notice. The web client shows announceNoMediaConnection only on an ICE failed state, and nothing on closed.
  • Nothing rejoins. Even connection_lost would only end the call with a notice; the client does not rejoin by itself.

H-L8-07's caps use the same trip, but only a hostile member can reach those, and only that member is hurt. The sender cap is the first one that an honest member, or another member (B2), can push a connection into.

What an honest member can reach:

  • Leaving the Talk view while in the call (useVisibleCameras, onBeforeUnmount) turns every camera off; coming back turns them all on again after 120 ms. In a Talk with 40 cameras that is 40 senders per return. 32 768 senders is about 820 returns: one every 35 s over 8 hours, or one a minute over a 13-hour day.
  • On top of that comes passive churn: other members' cameras, joins and resumes.

So the cap is unlikely to be hit by honest use, but not out of reach. The author's estimate of about 15 000 for a working day does not count view switching.

Fix: send the cap's close through the lost path (notifyLost, so the hub kicks with connection_lost and the seat is freed). Better still, let the client recover by itself: the web client already has a resume rejoin (rejoinAfterReconnect, resume: true, which the server treats as no leave/join within its grace time). An honest member then sees at most a short media gap.

BLOCKER B2: other members can drive a viewer's connection to the cap

Senders made because a publisher's track starts or resumes count against the viewer's cap. A publisher can stop its camera with one event and keep sending; the track is published again after resumeHoldoff (500 ms). Measured on loopback: about 1.9 new senders a second on every viewer's connection, using well under the event budget. So one member with ordinary rights to send video closes every other member's media connection after about 4.7 hours, again and again. Screen shares (for their viewers) and more accounts make it faster. With B1 the close is silent.

Without the cap, the same flapping now costs each viewer only about 8 MB over those 4.7 hours. The cap turns a bounded memory cost into a way to cut off other members' media.

Fix (either is enough with B1's recovery in place; without it, both are needed):

  • Bound the driver at the publisher: a budget for stop/resume cycles per source, for example a few per minute, or a holdoff that grows with repeated cycles. Then passive churn per viewer stays at honest levels.
  • Or count toward the cap only the senders the viewer's own subscriptions make, and bound the passive ones by the publisher's budget.

SHOULD-FIX

  • S1, metric reason. rtcp_ssrcs now also counts sender-cap closes. The doc and the metric comment explain this, so it is not wrong. But B1 and B2 make this cap one that honest members and other members' actions can reach, and operators need to tell those closes from hostile RTCP. Give it its own reason value (senders).

  • S2, CI cost. The new smoke takes 50.2 s under -race on this machine (the author measured 49 s): inside the 60 s smoke rule, but with only about 10 s to spare on a fast M-series Mac. CI runners are usually slower and may go past 60 s. The package under -race takes 94.0 s on the base e365c714d and 146.7 s on the tip (+52.6 s, +56%). CI runs it under -race twice (the unit job and the coverage/integration job), and internal/sfu is likely the slowest package, so that is about +50 s per job.

    Options:

    • Run the 2 000-switch slope in the nightly deep job and keep a cheaper smoke in PR CI. A deterministic check works without a heap slope, for example: no stopped sender stays reachable (a weak pointer or a finalizer on a stopped RTPSender).
    • Or keep the heap slope only in a non-race job.
  • S3, file size. internal/sfu/ingress.go is at 381 of 400 lines; the next change there will need a split.

NOTE

  • Lock order of sentTrack and trip in addSender: correct. sentTrack updates the counts under c.mu and releases it before trip. trip takes no lock: an atomic load of the connection, a CAS on closing, and PC.Close in its own goroutine. addSender runs under the peer's signalingMu; nothing on the close path takes signalingMu, so there is no deadlock. -race is clean.
  • Hostile cap-and-reconnect loop: each round costs at most about 8 MB, built over 100 to 200 s and freed when the connection closes. In my probe, 30 rounds left +2 MB and +0 goroutines once the offer watchdog's 10 s window had passed; about 1 MB per closed peer stays reachable until then, as documented in cycle_leak_test.go. Joining an existing Talk is not rate limited (only Create-a-Talk lobbies are), so the steady state is about 8 MB per hostile member at any time, plus the renegotiation CPU of ordinary toggling.
  • Commit message figures: "320 senders a second" is the event-budget upper bound; the measured pace on loopback was lower, so the per-second figures in SECURITY.md are conservative. Fine.

Revert checks

  • Trip in sentTrack removed: TestSenderCapClosesTheConnection fails ("timed out waiting for the connection closed past the cap").
  • bwe.go part of ef94fb6db reverted: TestSubscriptionSwitchesRetainNothing fails at +3 300 bytes per switch (bound 1 536).

Suites and gates (by exit code, on the tip)

  • go test -race -p 1 ./internal/sfu/: 0 (146.7 s). TestSubscriptionTogglesMakeQuietStreams did not fail.
  • internal/ws, internal/repo, internal/metrics: 0.
  • gofmt 0, go vet 0, go vet -tags=integration 0, make complexity-check 0, English check 0, make docs-check 0.
  • make bench-allocs: 0, at 0 allocs/op on all four benchmarks.
  • H-L8-05's known failure: unchanged (25 470 bytes per joiner, inverse assertion holds).
  • Commit types: test, fix, plus one merge. No attribution lines. The docs (SECURITY.md, doc/operations.md, the metric comment) match the code as written, apart from the points above.

Not examined

  • Real browsers. A browser's reaction to the server's DTLS close was not tested; the Pion client goes to closed, and the web client's only reaction, the ICE-failure notice, may not fire either.
  • Screen-share stop/resume cycles and camera layer moves (fit_px) as further B2 drivers. Read only: layered tracks switch layers inside one viewer track, so no new sender is expected from a layer move.
  • The desktop client.

Summary

  • Phase 2: STANDS, S3. Reproduced at 3.2 KB per switch. About 92% of it is H-L8-03's mechanism driven by a member. The residual of a few hundred bytes per sender is unbounded without the cap. The claimed numbers are corrected: about 2.2 MB a second per member before the fixes, about 87 KB a second after H-L8-03's.
  • Fix: MERGE AFTER FIXES.
    • B1: the cap closes the connection silently; no lost path, no notice, no recovery.
    • B2: other members' stop/resume churn drives every viewer to the cap (about 1.9 senders a second, about 4.7 h).
    • Should-fix: a reason value of its own for the metric, the smoke's -race cost in CI (+52.6 s per job), ingress.go near its size limit.

Round 2: tip a15ea9e18 (after f4e479574, the merge of fix-h-l8-07 at d77aa2893)

Verdict: MERGE. Both blockers are answered and every gate exits 0. Three items are left as should-fix and notes, none of them blocking.

B1: a cap close is no longer silent

peerStateChanged now hands a closed connection to closedPeer, which reports a cap close as lost. A sender-cap close is marked as a renewal (Peer.Renew):

  • the hub closes the member's socket with code 4002,
  • the seat is kept for the grace period,
  • the client reconnects after 1 s and resumes with voice_join, resume: true.

The abuse caps take the lost path and end the voice with voice_kicked connection_lost.

The author's tests for this pass: TestSenderCapRenewsTheConnection, TestUnhandledSRTCPStreamsAreCapped, TestSenderCapRenewsTheMembersConnection and TestAbuseCapEndsTheMembersVoice.

What a renewal costs the member, for about one to two seconds. I read the socket's unregister path (lifecycle.go, forget) and the client's send path:

  • Media: a gap of about 1 to 2 s. The seat stays, and no voice_state_update is sent (the author's ws test checks this).
  • Presence: when the renewed socket is the member's only connection, forget broadcasts presence_update offline at once, and online follows on reconnect. Other members see a flicker. The chosen status (dnd, focus) survives: the handshake restores it from the account. This is what every dropped socket already does (NOTE R2-1).
  • Events sent during the gap: sendSocketEvent drops them silently (typing, speaking, a subscription change). Messages, edits, pins and uploads go over HTTP and are not affected. The bot gateway is a separate endpoint and is not touched.
  • Per-user gates: on the last connection, the voice, stage and reaction gates and the user's event budgets are reset. A renewal needs at least about 100 s of sender making, so this is negligible.

Old web clients and the desktop client:

  • web/src/stores/socket.ts is unchanged since v3.0.9. Any close code except 4001 reconnects after 1 s, and 3.0.9's App.vue already runs rejoinAfterReconnect on reconnectCount, with voice_join, resume: true. So a 3.0.9 client handles 4002 correctly without knowing it.
  • Desktop repository (read only, d5cde742): the Linux client's socket proxy (linux/src/core/sockets.ts) and the experimental one (web_ws.rs) pass the instance's close code through to the page, which runs the same web socket store. So it behaves the same.
  • Not tested in a real browser or a real desktop build.

API stability:

  • doc/api-stability.md does not list close codes. A new close code is additive under its rule for enum values ("clients must ignore what they do not know"), and old clients do in fact treat it as an ordinary drop.
  • It is documented in the WebSocket description (lifecycle.go), and api/openapi.json was regenerated: make openapi-check exits 0.
  • NOTE R2-2: the bot gateway already uses 4002 with another meaning (a bad frame, doc/bots.md). That is legal on a separate endpoint, but one line in doc/bots.md or the WebSocket description saying the codes are per endpoint would prevent confusion.

Test seam. Peer.TripCap is defined only in internal/sfu/testhooks.go. A search of the whole tree finds no caller outside _test.go files. It is an exported method in a production file, which is the existing pattern of that file, so it is compiled into the server, but no handler or other package calls it.

B2: the resume budget bounds what others can drive

Each track now has a resume budget: 10 quick resumes, then one per 10 s.

  • Measured on loopback: 5 publishers stopping and resuming their cameras as fast as the budget allows for 60 s made 78 senders on one viewer, 15.6 per source. That is exactly the burst of 10 plus 6 refills, so the steady state is 0.1 a second per source. TestStopResumeBudget and TestTrackPastItsResumeBudgetStaysStopped pass.
  • Worst case asked for (50 publishers, camera and screen each, within the budget): about 10 new senders a second per viewer watching all screens, about 5 a second for cameras only, plus a one-time burst of 10 per source. That reaches the cap after about 53 minutes, or about 1.8 hours for cameras only. The result is a renewal (a 1 to 2 s gap and a presence flicker), not an outage.
  • One hostile member with two sources: 0.2 a second, about 45 hours to the cap.
  • SHOULD-FIX R2-3: the budget lives on the TrackInfo. A capture that is replaced (a new track for the same source) or a publisher that rejoins starts again with a full burst. Rejoins are limited by voice_join (about one a second, burst 3) and are H-L8-05's join-churn path. Keying the budget by publisher and source in the room would make it survive replacement and rejoin within the room's life.

Honest use:

  • Camera visibility (cameraVisibility.ts, useVisibleCameras) only changes the viewer's subscriptions (webrtc_subscribe); it never stops the publisher's track.
  • Mute and push-to-talk set track.enabled (audioSettings.ts) and send no stop event. Audio is never unpublished by a client stop.
  • Only a camera switched off by hand (camera.ts, webrtc_camera_stop) and the end of a share (screen.ts) spend the budget.

Past the budget: viewers see the publisher's camera tile without a picture for up to 10 s (the media state still says the camera is on). The publisher sees its own preview and gets no notice. Acceptable for something that needs more than 10 camera toggles in quick succession (NOTE R2-4: a hint to the publisher would make it explainable).

Should-fix from round 1

  • S1: the metric reason is now senders, and rtcp_ssrcs keeps the abuse caps. The doc and the metric comment match. Closed.
  • S2: the 2 000-switch slope runs only with SFU_SUB_SLOPE=1, in the nightly sim-sweep.yml sfu job. The PR smoke is now TestStoppedSendersAreReleased: 0.15 s, and reliable both ways. With ef94fb6db's bwe.go change reverted it failed 20 of 20 runs; with the fix it passed 20 of 20. Under -race, internal/sfu takes 101.7 s (94.0 s on the round 1 base, 146.7 s on the previous tip). Closed.
  • S3: the cap code moved to transport.go; ingress.go is down to 329 lines. NOTE R2-5: room_tracks.go is at 398 and room_attach.go at 396 of 400 lines, so the next change in either needs a split.

Suites and gates (by exit code, on a15ea9e18)

  • go test -race -p 1 ./internal/sfu/ ./internal/ws/: 0 (sfu 101.7 s, ws 49.9 s). TestSubscriptionTogglesMakeQuietStreams did not fail.
  • gofmt 0, go vet 0, go vet -tags=integration 0 (sfu, ws, metrics).
  • make complexity-check 0, English check 0, make docs-check 0, make openapi-check 0.
  • internal/repo and internal/metrics tests 0.
  • make bench-allocs 0, at 0 allocs/op on all four benchmarks.
  • make fuzz-smoke (FUZZTIME=5s): exit 2, but not from this fix. FuzzCleanDomain in internal/automod found an input where cleanDomain and hostOf disagree. internal/automod is unchanged against origin/main, and the same corpus case fails on origin/main (f58bc3fc6), so it is pre-existing. The corpus file is kept in reviews/fixreview-H-L8-09-fuzzcase/ for the coordinator to triage. Every other fuzz target passed.

Not examined

  • Real browsers, and a real desktop build, reacting to close code 4002.
  • A screen share past its budget, beyond the camera case.

Round 3: tip bbfec1723 (after e19560f05, the merge of fix-h-l9-01)

Verdict: MERGE. Round 2's should-fix R2-3 is closed and every gate exits 0. Nothing new blocks.

The resume budget per publisher and source

Room.resumeBudgets is keyed by publisher and source and is guarded by r.mu. It is reached only through unpublish → stopSourceTracksLocked → stopUntilLocked, under the room's write lock, and it reads only r.peers and the map. No other lock is taken, so the lock order is r.mu only.

  • One stop event spends one resume, however many tracks it ends (layers, the share's sound): resumeAt is computed once per stop.
  • The clamp at −1 does not allow faster cycling. A stop acts only on live tracks, and over budget a track resumes only once the balance has refilled from −1 to 0, 10 s later. So the steady rate through resumes stays at one per 10 s per source, as before. Without the clamp, repeated stops could only push the balance lower than −1 if new captures of the same source kept arriving while one was held. The clamp caps that wait at 10 s; it gives no extra resumes.
  • Rejoins and replaced captures no longer reset the budget. An entry of a publisher who left is deleted only once it has refilled to full, so forgetting it never grants more than the refill would have. TestResumeBudgetOutlivesCapturesAndRejoins passes. Revert check: with a fresh budget per stop (the key built from a new UUID) it fails, with 200 quick resumes against at most 17.
  • Measured again on the live path: 5 publishers stopping and resuming their cameras as fast as allowed for 60 s made 77 senders on one viewer, 15.4 per source: the burst of 10 plus about 6 refills, 0.1 a second per source steady. The map held 5 entries. The worst case asked for (50 publishers, camera and screen) is unchanged from round 2: about 10 senders a second per viewer, the cap after about 53 minutes, then a renewal.

Bound of the map when nobody stops again

Entries are only created by a stop: screen and camera by the publisher's own stop, and the microphone through a permission revoke (RevokeUserSource). They are only deleted inside a later stop in the same room. So if nobody ever stops again, entries stay until the room is closed, which happens when it is empty.

  • Bound: at most 3 entries per distinct member who stopped a source while the room stayed open (one per source), about 120 to 150 bytes each (a 16-byte UUID and a string header as the key, a pointer, and a resumeBudget of 32 bytes, plus map overhead).
  • Example: a room that never empties for months, with 10 000 distinct members each stopping two sources, holds about 20 000 entries, roughly 3 MB. One account can never hold more than 3 entries.
  • Cost of the sweep: the sweep inside stopUntilLocked is linear in the map, but stops themselves are budgeted (about 0.1 a second per source), so it stays negligible.

NOTE R3-1: acceptable as is. If rooms that never close matter, a sweep on peer removal would bound the map by the members seated in the last 110 s, not by those ever seated.

Steady clock (the fix-h-l9-01 merge)

The deadline is steadyNanos(now + wait), and resumeStoppedTrack compares it with steadyNanos(layerNow()). The budget's at and leftAt work on layerNow() values. All of the lane 9 clock-step tests pass on the tip, including TestClockStepBackResumesAStoppedCamera and TestClockStepForwardKeepsTheResumeHoldoff.

Docs

doc/bots.md and the WebSocket description now separate the member socket's 4002 (media renewed) from the bot gateway's 4002 (a bad frame). NOTE R2-2 is closed. make openapi-check exits 0.

Suites and gates (by exit code, on bbfec1723)

  • go test -race -p 1 ./internal/sfu/ ./internal/ws/: 0 (sfu 103.5 s, ws 49.4 s). TestSubscriptionTogglesMakeQuietStreams did not fail.
  • gofmt 0, go vet 0, go vet -tags=integration 0.
  • make complexity-check 0, English check 0, make docs-check 0, make openapi-check 0, internal/repo tests 0.
  • make bench-allocs 0, at 0 allocs/op on all four benchmarks.
  • File sizes: sources.go 167, room_tracks.go 379 (down from 398), room_attach.go 396 (NOTE R2-5 still applies to this one), sfu.go 328.
  • Commit type fix, no attribution lines.

Not examined (NOTE R3-2, for a follow-up)

A new capture of the same source, meaning a new remote track rather than a resume, makes a sender on every viewer outside the resume budget. Its rate is bounded by renegotiation, and with the sender-cap renewal the worst effect is a short media gap. I did not measure it.

Something wrong on this page? Tell us.