Phase 2 and fix review: H-L8-09 (subscription switches keep memory per switch)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: independent (I wrote neither the finding nor the fix).
Finding: findings/H-L8-09.md (S3 claimed).
Fix branch: origin/msch128/fix-h-l8-09, on origin/msch128/fix-h-l8-07 at
e365c714d:
1c787fdc2: test, a known failure ininternal/sfu/subscription_cycle_test.go.ff88d92aa: merge ofmsch128/fix-h-l8-03, the already reviewed fix that forgets the bandwidth estimator's writer of a stopped sender.714225dd4: fix, a media connection is closed past 32 768 senders made on it; the test now asserts a bound of 1.5 KB per switch.
All runs were on loopback, in temporary worktrees that are now removed, with
nice -n 10 and -p 1, one heavy job at a time, at a load under 12. No audio
device, camera or microphone was used.
Phase 2 verdict: STANDS (S3), with corrected figures. Not a duplicate. Fix verdict: MERGE AFTER FIXES (two blockers).
Phase 2
Reproduction
On 1c787fdc2, go test ./internal/sfu/ -run TestSubscriptionSwitches -v
measured +3 175 bytes per switch over 1 000 switches. Goroutines +0, offer
26 984 → 26 984 bytes. The inverse assertion holds.
Is it documented?
No. Before the fix, SECURITY.md (media connection limits) said nothing about per-sender state kept for the life of a connection.
Is it a duplicate of H-L8-03 or H-L8-05?
H-L8-03. At ff88d92aa (H-L8-03's fix merged, no cap yet), the same test
at 4 000 switches measures +265 bytes per switch, down from 3 175. So
about 92% of the measured retention is H-L8-03's defect: the stopped sender's
writer chain, pinned by the pacer. H-L8-09's headline figure is H-L8-03 seen
from the member's side.
H-L8-09 is still a finding of its own, for two reasons:
- Reach. H-L8-03 is rated S4: passive churn, "no attacker needed". H-L8-09 is an ordinary member driving the same retention on purpose, at the speed of its own events.
- What is left. H-L8-03's fix does not touch the residual of a few hundred bytes per sender (Pion's per-SSRC state and the rest). Without the cap it is unbounded for the life of the connection and drivable by the member.
H-L8-05 is a different path: state kept on a viewer's connection per member who joined and left. Its known failure is unchanged on the fix tip: 25 470 bytes per joiner at 1 200 rounds, inverse assertion still holds.
Severity
S3 stands. Ordinary members can bypass the per-connection bounds SECURITY.md states, in both states:
- Before any fix: with camera batches of 16 (
SubscribeCameras, the path of the batch event), I measured 6 843 bytes per sender. At the WebSocket budget of 40 events a second (20 off/on toggles, 320 senders a second) that is about 2.2 MB a second per member. - With H-L8-03's fix only: about 273 bytes per sender, both ends of the connection (measured on the tip below, which differs from the H-L8-03-only state by the cap alone). At the event budget that is about 87 KB a second per member, unbounded, about 1 GB in 3.4 hours, times the number of members.
Corrections to the claim:
- "16.4 KB per event" includes warm-up. The steady figure is about 3.2 KB per switch on a screen share, about 6.8 KB per sender.
- "About 10 MB a second" with camera batches is about 4.5 times too high. Measured and scaled to the event budget it is about 2.2 MB a second.
Fix review (tip 714225dd4)
What still grows, measured
I diffed heap profiles taken after two forced GCs (MemProfileRate 1024) and
fitted the heap in use per switch.
- Screen share (the fix's own test, 2 000 switches): +118 to +430 bytes per
switch, depending on run and load (118 under
-race). Goroutines +0, offer flat. - Cameras, not measured by the author:
- 4 publishers, 600 toggles, 2 400 senders: +836 bytes per sender, with the curve flattening.
- 16 publishers (full batches), 300 toggles, 4 800 senders: +273 bytes per sender for both ends. That matches the author's 160 to 240 bytes on the server side.
- The profile diff on the server side shows no stopped sender and no writer chain. What remains is SDP and codec parse objects of the current descriptions (live in both snapshots, noise) and a small SRTP/SRTCP share.
- In rapid toggles most senders never encrypt a packet before they are stopped, so the SRTP state per sender is below the author's 156 bytes. Either way, the cost is a few hundred bytes per sender, and the cap bounds it.
- Harness note: in my first camera runs the publishers' own NACK send
history filled up during the run (
rtpbuffercopies under the test client'spublishloop). That is test-client growth, not the SFU. Filling both tracks' histories first removes it. The fix's screen-share test already does this (fillSendHistory).
Pace
Measured on the tip with batches of 16 (in process, no event budget) on this machine, at a load of about 10: about 164 senders a second, limited by the renegotiation round trip. Before the fix the same probe on a quieter machine ran at about 970 a second. The author's figure, 320 senders a second at the event budget, is the right upper bound to plan with. At that pace one member reaches the cap in about 100 to 200 seconds.
BLOCKER B1: the cap closes the connection silently
trip calls PC.Close(). peerStateChanged treats the closed state with
removePeer alone; only the failed state goes through dropLostPeer →
notifyLost. I checked it with a probe: after the sender cap tripped, the
peer's OnLost callback did not fire within 15 s, and the test client's
connection went to closed.
So the hub is never told:
- The member keeps the call seat, and the others see them in the call.
- The member gets no
voice_kickedand no notice. The web client showsannounceNoMediaConnectiononly on an ICEfailedstate, and nothing onclosed. - Nothing rejoins. Even
connection_lostwould only end the call with a notice; the client does not rejoin by itself.
H-L8-07's caps use the same trip, but only a hostile member can reach those,
and only that member is hurt. The sender cap is the first one that an honest
member, or another member (B2), can push a connection into.
What an honest member can reach:
- Leaving the Talk view while in the call (
useVisibleCameras,onBeforeUnmount) turns every camera off; coming back turns them all on again after 120 ms. In a Talk with 40 cameras that is 40 senders per return. 32 768 senders is about 820 returns: one every 35 s over 8 hours, or one a minute over a 13-hour day. - On top of that comes passive churn: other members' cameras, joins and resumes.
So the cap is unlikely to be hit by honest use, but not out of reach. The author's estimate of about 15 000 for a working day does not count view switching.
Fix: send the cap's close through the lost path (notifyLost, so the hub
kicks with connection_lost and the seat is freed). Better still, let the
client recover by itself: the web client already has a resume rejoin
(rejoinAfterReconnect, resume: true, which the server treats as no
leave/join within its grace time). An honest member then sees at most a short
media gap.
BLOCKER B2: other members can drive a viewer's connection to the cap
Senders made because a publisher's track starts or resumes count against the
viewer's cap. A publisher can stop its camera with one event and keep
sending; the track is published again after resumeHoldoff (500 ms). Measured
on loopback: about 1.9 new senders a second on every viewer's connection,
using well under the event budget. So one member with ordinary rights to send
video closes every other member's media connection after about 4.7 hours,
again and again. Screen shares (for their viewers) and more accounts make it
faster. With B1 the close is silent.
Without the cap, the same flapping now costs each viewer only about 8 MB over those 4.7 hours. The cap turns a bounded memory cost into a way to cut off other members' media.
Fix (either is enough with B1's recovery in place; without it, both are needed):
- Bound the driver at the publisher: a budget for stop/resume cycles per source, for example a few per minute, or a holdoff that grows with repeated cycles. Then passive churn per viewer stays at honest levels.
- Or count toward the cap only the senders the viewer's own subscriptions make, and bound the passive ones by the publisher's budget.
SHOULD-FIX
S1, metric reason.
rtcp_ssrcsnow also counts sender-cap closes. The doc and the metric comment explain this, so it is not wrong. But B1 and B2 make this cap one that honest members and other members' actions can reach, and operators need to tell those closes from hostile RTCP. Give it its own reason value (senders).S2, CI cost. The new smoke takes 50.2 s under
-raceon this machine (the author measured 49 s): inside the 60 s smoke rule, but with only about 10 s to spare on a fast M-series Mac. CI runners are usually slower and may go past 60 s. The package under-racetakes 94.0 s on the basee365c714dand 146.7 s on the tip (+52.6 s, +56%). CI runs it under-racetwice (the unit job and the coverage/integration job), andinternal/sfuis likely the slowest package, so that is about +50 s per job.Options:
- Run the 2 000-switch slope in the nightly deep job and keep a cheaper smoke
in PR CI. A deterministic check works without a heap slope, for example:
no stopped sender stays reachable (a weak pointer or a finalizer on a
stopped
RTPSender). - Or keep the heap slope only in a non-race job.
- Run the 2 000-switch slope in the nightly deep job and keep a cheaper smoke
in PR CI. A deterministic check works without a heap slope, for example:
no stopped sender stays reachable (a weak pointer or a finalizer on a
stopped
S3, file size.
internal/sfu/ingress.gois at 381 of 400 lines; the next change there will need a split.
NOTE
- Lock order of sentTrack and trip in addSender: correct.
sentTrackupdates the counts underc.muand releases it beforetrip.triptakes no lock: an atomic load of the connection, a CAS onclosing, andPC.Closein its own goroutine.addSenderruns under the peer'ssignalingMu; nothing on the close path takessignalingMu, so there is no deadlock.-raceis clean. - Hostile cap-and-reconnect loop: each round costs at most about 8 MB,
built over 100 to 200 s and freed when the connection closes. In my probe,
30 rounds left +2 MB and +0 goroutines once the offer watchdog's 10 s window
had passed; about 1 MB per closed peer stays reachable until then, as
documented in
cycle_leak_test.go. Joining an existing Talk is not rate limited (only Create-a-Talk lobbies are), so the steady state is about 8 MB per hostile member at any time, plus the renegotiation CPU of ordinary toggling. - Commit message figures: "320 senders a second" is the event-budget upper bound; the measured pace on loopback was lower, so the per-second figures in SECURITY.md are conservative. Fine.
Revert checks
- Trip in
sentTrackremoved:TestSenderCapClosesTheConnectionfails ("timed out waiting for the connection closed past the cap"). bwe.gopart ofef94fb6dbreverted:TestSubscriptionSwitchesRetainNothingfails at +3 300 bytes per switch (bound 1 536).
Suites and gates (by exit code, on the tip)
go test -race -p 1 ./internal/sfu/: 0 (146.7 s).TestSubscriptionTogglesMakeQuietStreamsdid not fail.internal/ws,internal/repo,internal/metrics: 0.- gofmt 0,
go vet0,go vet -tags=integration0,make complexity-check0, English check 0,make docs-check0. make bench-allocs: 0, at 0 allocs/op on all four benchmarks.- H-L8-05's known failure: unchanged (25 470 bytes per joiner, inverse assertion holds).
- Commit types:
test,fix, plus one merge. No attribution lines. The docs (SECURITY.md, doc/operations.md, the metric comment) match the code as written, apart from the points above.
Not examined
- Real browsers. A browser's reaction to the server's DTLS close was not
tested; the Pion client goes to
closed, and the web client's only reaction, the ICE-failure notice, may not fire either. - Screen-share stop/resume cycles and camera layer moves (
fit_px) as further B2 drivers. Read only: layered tracks switch layers inside one viewer track, so no new sender is expected from a layer move. - The desktop client.
Summary
- Phase 2: STANDS, S3. Reproduced at 3.2 KB per switch. About 92% of it is H-L8-03's mechanism driven by a member. The residual of a few hundred bytes per sender is unbounded without the cap. The claimed numbers are corrected: about 2.2 MB a second per member before the fixes, about 87 KB a second after H-L8-03's.
- Fix: MERGE AFTER FIXES.
- B1: the cap closes the connection silently; no lost path, no notice, no recovery.
- B2: other members' stop/resume churn drives every viewer to the cap (about 1.9 senders a second, about 4.7 h).
- Should-fix: a reason value of its own for the metric, the smoke's
-racecost in CI (+52.6 s per job),ingress.gonear its size limit.
Round 2: tip a15ea9e18 (after f4e479574, the merge of fix-h-l8-07 at d77aa2893)
Verdict: MERGE. Both blockers are answered and every gate exits 0. Three items are left as should-fix and notes, none of them blocking.
B1: a cap close is no longer silent
peerStateChanged now hands a closed connection to closedPeer, which reports
a cap close as lost. A sender-cap close is marked as a renewal
(Peer.Renew):
- the hub closes the member's socket with code 4002,
- the seat is kept for the grace period,
- the client reconnects after 1 s and resumes with
voice_join,resume: true.
The abuse caps take the lost path and end the voice with voice_kicked
connection_lost.
The author's tests for this pass: TestSenderCapRenewsTheConnection,
TestUnhandledSRTCPStreamsAreCapped, TestSenderCapRenewsTheMembersConnection
and TestAbuseCapEndsTheMembersVoice.
What a renewal costs the member, for about one to two seconds. I read the
socket's unregister path (lifecycle.go, forget) and the client's send
path:
- Media: a gap of about 1 to 2 s. The seat stays, and no
voice_state_updateis sent (the author's ws test checks this). - Presence: when the renewed socket is the member's only connection,
forgetbroadcastspresence_updateofflineat once, andonlinefollows on reconnect. Other members see a flicker. The chosen status (dnd, focus) survives: the handshake restores it from the account. This is what every dropped socket already does (NOTE R2-1). - Events sent during the gap:
sendSocketEventdrops them silently (typing, speaking, a subscription change). Messages, edits, pins and uploads go over HTTP and are not affected. The bot gateway is a separate endpoint and is not touched. - Per-user gates: on the last connection, the voice, stage and reaction gates and the user's event budgets are reset. A renewal needs at least about 100 s of sender making, so this is negligible.
Old web clients and the desktop client:
web/src/stores/socket.tsis unchanged since v3.0.9. Any close code except 4001 reconnects after 1 s, and 3.0.9'sApp.vuealready runsrejoinAfterReconnectonreconnectCount, withvoice_join,resume: true. So a 3.0.9 client handles 4002 correctly without knowing it.- Desktop repository (read only,
d5cde742): the Linux client's socket proxy (linux/src/core/sockets.ts) and the experimental one (web_ws.rs) pass the instance's close code through to the page, which runs the same web socket store. So it behaves the same. - Not tested in a real browser or a real desktop build.
API stability:
doc/api-stability.mddoes not list close codes. A new close code is additive under its rule for enum values ("clients must ignore what they do not know"), and old clients do in fact treat it as an ordinary drop.- It is documented in the WebSocket description (
lifecycle.go), andapi/openapi.jsonwas regenerated:make openapi-checkexits 0. - NOTE R2-2: the bot gateway already uses 4002 with another meaning (a bad
frame,
doc/bots.md). That is legal on a separate endpoint, but one line indoc/bots.mdor the WebSocket description saying the codes are per endpoint would prevent confusion.
Test seam. Peer.TripCap is defined only in internal/sfu/testhooks.go.
A search of the whole tree finds no caller outside _test.go files. It is an
exported method in a production file, which is the existing pattern of that
file, so it is compiled into the server, but no handler or other package
calls it.
B2: the resume budget bounds what others can drive
Each track now has a resume budget: 10 quick resumes, then one per 10 s.
- Measured on loopback: 5 publishers stopping and resuming their cameras as
fast as the budget allows for 60 s made 78 senders on one viewer, 15.6 per
source. That is exactly the burst of 10 plus 6 refills, so the steady state
is 0.1 a second per source.
TestStopResumeBudgetandTestTrackPastItsResumeBudgetStaysStoppedpass. - Worst case asked for (50 publishers, camera and screen each, within the budget): about 10 new senders a second per viewer watching all screens, about 5 a second for cameras only, plus a one-time burst of 10 per source. That reaches the cap after about 53 minutes, or about 1.8 hours for cameras only. The result is a renewal (a 1 to 2 s gap and a presence flicker), not an outage.
- One hostile member with two sources: 0.2 a second, about 45 hours to the cap.
- SHOULD-FIX R2-3: the budget lives on the
TrackInfo. A capture that is replaced (a new track for the same source) or a publisher that rejoins starts again with a full burst. Rejoins are limited byvoice_join(about one a second, burst 3) and are H-L8-05's join-churn path. Keying the budget by publisher and source in the room would make it survive replacement and rejoin within the room's life.
Honest use:
- Camera visibility (
cameraVisibility.ts,useVisibleCameras) only changes the viewer's subscriptions (webrtc_subscribe); it never stops the publisher's track. - Mute and push-to-talk set
track.enabled(audioSettings.ts) and send no stop event. Audio is never unpublished by a client stop. - Only a camera switched off by hand (
camera.ts,webrtc_camera_stop) and the end of a share (screen.ts) spend the budget.
Past the budget: viewers see the publisher's camera tile without a picture for up to 10 s (the media state still says the camera is on). The publisher sees its own preview and gets no notice. Acceptable for something that needs more than 10 camera toggles in quick succession (NOTE R2-4: a hint to the publisher would make it explainable).
Should-fix from round 1
- S1: the metric reason is now
senders, andrtcp_ssrcskeeps the abuse caps. The doc and the metric comment match. Closed. - S2: the 2 000-switch slope runs only with
SFU_SUB_SLOPE=1, in the nightlysim-sweep.ymlsfu job. The PR smoke is nowTestStoppedSendersAreReleased: 0.15 s, and reliable both ways. Withef94fb6db'sbwe.gochange reverted it failed 20 of 20 runs; with the fix it passed 20 of 20. Under-race,internal/sfutakes 101.7 s (94.0 s on the round 1 base, 146.7 s on the previous tip). Closed. - S3: the cap code moved to
transport.go;ingress.gois down to 329 lines. NOTE R2-5:room_tracks.gois at 398 androom_attach.goat 396 of 400 lines, so the next change in either needs a split.
Suites and gates (by exit code, on a15ea9e18)
go test -race -p 1 ./internal/sfu/ ./internal/ws/: 0 (sfu 101.7 s, ws 49.9 s).TestSubscriptionTogglesMakeQuietStreamsdid not fail.- gofmt 0,
go vet0,go vet -tags=integration0 (sfu, ws, metrics). make complexity-check0, English check 0,make docs-check0,make openapi-check0.internal/repoandinternal/metricstests 0.make bench-allocs0, at 0 allocs/op on all four benchmarks.make fuzz-smoke(FUZZTIME=5s): exit 2, but not from this fix.FuzzCleanDomainininternal/automodfound an input wherecleanDomainandhostOfdisagree.internal/automodis unchanged againstorigin/main, and the same corpus case fails onorigin/main(f58bc3fc6), so it is pre-existing. The corpus file is kept inreviews/fixreview-H-L8-09-fuzzcase/for the coordinator to triage. Every other fuzz target passed.
Not examined
- Real browsers, and a real desktop build, reacting to close code 4002.
- A screen share past its budget, beyond the camera case.
Round 3: tip bbfec1723 (after e19560f05, the merge of fix-h-l9-01)
Verdict: MERGE. Round 2's should-fix R2-3 is closed and every gate exits 0. Nothing new blocks.
The resume budget per publisher and source
Room.resumeBudgets is keyed by publisher and source and is guarded by
r.mu. It is reached only through unpublish → stopSourceTracksLocked →
stopUntilLocked, under the room's write lock, and it reads only r.peers
and the map. No other lock is taken, so the lock order is r.mu only.
- One stop event spends one resume, however many tracks it ends (layers,
the share's sound):
resumeAtis computed once per stop. - The clamp at −1 does not allow faster cycling. A stop acts only on live tracks, and over budget a track resumes only once the balance has refilled from −1 to 0, 10 s later. So the steady rate through resumes stays at one per 10 s per source, as before. Without the clamp, repeated stops could only push the balance lower than −1 if new captures of the same source kept arriving while one was held. The clamp caps that wait at 10 s; it gives no extra resumes.
- Rejoins and replaced captures no longer reset the budget. An entry of a
publisher who left is deleted only once it has refilled to full, so
forgetting it never grants more than the refill would have.
TestResumeBudgetOutlivesCapturesAndRejoinspasses. Revert check: with a fresh budget per stop (the key built from a new UUID) it fails, with 200 quick resumes against at most 17. - Measured again on the live path: 5 publishers stopping and resuming their cameras as fast as allowed for 60 s made 77 senders on one viewer, 15.4 per source: the burst of 10 plus about 6 refills, 0.1 a second per source steady. The map held 5 entries. The worst case asked for (50 publishers, camera and screen) is unchanged from round 2: about 10 senders a second per viewer, the cap after about 53 minutes, then a renewal.
Bound of the map when nobody stops again
Entries are only created by a stop: screen and camera by the publisher's own
stop, and the microphone through a permission revoke (RevokeUserSource).
They are only deleted inside a later stop in the same room. So if nobody ever
stops again, entries stay until the room is closed, which happens when it is
empty.
- Bound: at most 3 entries per distinct member who stopped a source while
the room stayed open (one per source), about 120 to 150 bytes each (a
16-byte UUID and a string header as the key, a pointer, and a
resumeBudgetof 32 bytes, plus map overhead). - Example: a room that never empties for months, with 10 000 distinct members each stopping two sources, holds about 20 000 entries, roughly 3 MB. One account can never hold more than 3 entries.
- Cost of the sweep: the sweep inside
stopUntilLockedis linear in the map, but stops themselves are budgeted (about 0.1 a second per source), so it stays negligible.
NOTE R3-1: acceptable as is. If rooms that never close matter, a sweep on peer removal would bound the map by the members seated in the last 110 s, not by those ever seated.
Steady clock (the fix-h-l9-01 merge)
The deadline is steadyNanos(now + wait), and resumeStoppedTrack compares
it with steadyNanos(layerNow()). The budget's at and leftAt work on
layerNow() values. All of the lane 9 clock-step tests pass on the tip,
including TestClockStepBackResumesAStoppedCamera and
TestClockStepForwardKeepsTheResumeHoldoff.
Docs
doc/bots.md and the WebSocket description now separate the member socket's
4002 (media renewed) from the bot gateway's 4002 (a bad frame). NOTE R2-2 is
closed. make openapi-check exits 0.
Suites and gates (by exit code, on bbfec1723)
go test -race -p 1 ./internal/sfu/ ./internal/ws/: 0 (sfu 103.5 s, ws 49.4 s).TestSubscriptionTogglesMakeQuietStreamsdid not fail.- gofmt 0,
go vet0,go vet -tags=integration0. make complexity-check0, English check 0,make docs-check0,make openapi-check0,internal/repotests 0.make bench-allocs0, at 0 allocs/op on all four benchmarks.- File sizes:
sources.go167,room_tracks.go379 (down from 398),room_attach.go396 (NOTE R2-5 still applies to this one),sfu.go328. - Commit type
fix, no attribution lines.
Not examined (NOTE R3-2, for a follow-up)
A new capture of the same source, meaning a new remote track rather than a resume, makes a sender on every viewer outside the resume budget. Its rate is bounded by renegotiation, and with the sender-cap renewal the worst effect is a short media gap. I did not measure it.