Fix review: H-L8-03 (bandwidth estimator pacer never forgets a stopped stream)
From mnema-talk v3.3.13, rendered on this site.
Fix: branch msch128/fix-h-l8-03, commit ef94fb6db ("fix: forget a stopped stream in the bandwidth estimator's pacer"), on the lane 8 test commit 2e28b6f6c. Reviewed: the diff of ef94fb6db only (internal/sfu/bwe.go +25/-2, internal/sfu/bwe_writers_leak_test.go). Read alongside: the finding, the repro and refute reports, internal/sfu/{bwe.go, sfu_setup.go, room_signal.go, room_attach.go, viewer_track.go, nack_responder.go}, pion/webrtc v4.2.26 (rtpsender.go, peerconnection.go, rtptransceiver.go, track_local_static.go) and pion/interceptor v0.1.51 (chain.go, pkg/gcc, internal/cc, pkg/twcc, pkg/report, pkg/stats, pkg/nack). Temp git worktree, removed afterwards.
Verdict: MERGE
No BLOCKER, no SHOULD-FIX. Three NOTEs.
Does it remove the cause
Yes. bweInterceptor.UnbindLocalStream (bwe.go:101-103) calls passPacer.remove (bwe.go:164-171),
which deletes info.SSRC and, only when non-zero, info.SSRCRetransmission under the pacer's write
lock. This mirrors BindLocalStream exactly (bwe.go:88-96: est.AddStream registers info.SSRC
through gcc, the RTX alias is added only when SSRCRetransmission != 0). FEC SSRCs were never
registered, so there is nothing to forget for them.
Bind and unbind receive the same values in production. Pion binds once per encoding in
RTPSender.Send with &trackEncoding.streamInfo (rtpsender.go:350-366) and unbinds the same pointer
in RTPSender.Stop (rtpsender.go:400). The SFU stops viewer senders only through
PC.RemoveTrack (room_signal.go:181, 225), which calls sender.Stop() (peerconnection.go:2536);
PeerConnection.Close stops every sender the same way. The interceptor chain calls every
interceptor's UnbindLocalStream (chain.go:47-51), so ours is reached.
Simulcast: viewer senders are created only by PC.AddTrack (room_signal.go:215) with one track, so
each has exactly one encoding and one bind. Layer groups give the viewer one track and switch the
source inside it (layer_viewers.go), so a layer switch is no bind and no unbind. Even a multi-encoding
sender would be symmetric: Send binds and Stop unbinds once per encoding, each with its own
StreamInfo.
gcc: SendSideBWE.AddStream (send_side_bwe.go:153-180) wraps the writer in a closure that is stored
only in our pacer (e.pacer.AddStream) and returns the pacer itself. SendSideBWE holds no stream map
(its fields: pacer, loss and delay controllers, feedback adapter, bitrate state); the feedback
adapter's history is a fixed 250-entry list of (SSRC, sequence) send records with no writer
reference. After the delete nothing in gcc references the stopped sender's writer chain, so gcc can
neither call it nor keep it alive. The other registered interceptors: the transport-cc stamp keeps a
single counter (no per-stream state); Pion's report sender, stats and NACK responder implement
UnbindLocalStream; the NACK generator is a remote-stream interceptor with UnbindRemoteStream; ours
(nackResponder) unbinds its stream (nack_responder.go:134-143). The commit message's inventory is
correct.
Through a real Pion PeerConnection (the lead's extra point)
Own temporary probe test (not committed), using the package's loopback clients and the SFU's real
registry (NewSFU -> buildAPI -> buildInterceptors). It wraps newEstimator to capture each
connection's *gcc.SendSideBWE and reaches its pacer through AddStream (which returns the pacer),
removing the sentinel it added. Publisher: screen (VP8, with RTX) and microphone (Opus, no RTX).
Viewer: joins, then 15 times Room.Subscribe(screen, on) and (screen, off), waiting each time for
the renegotiation; then RemovePeer of the publisher.
After every step the set of keys in the viewer's passPacer.writers was compared with the set
{encoding SSRC, RTX SSRC} of the viewer's live senders (GetSenders()[i].GetParameters()):
initial: pacer [mic] senders [mic]
screen on: pacer [mic, scr, scr-rtx] senders [mic, scr, scr-rtx]
after 15 on/off: pacer [mic] senders [mic]
after publisher left: pacer []
The sets were equal after every step, so the unbind gets the same SSRC and RTX SSRC as the bind, and stopping the screen sender never removed the microphone's entry. The captured SFU log had zero "unknown stream" lines. With bwe.go reverted to 2e28b6f6c the same probe times out at the first "screen off" (the screen's two entries stay).
Can one sender's unbind remove another live sender's entry
- Different SSRCs: no (own probe above, and the author's TestPassPacerUnbindKeepsOtherStreams).
- RTX zero:
removeskips it, so an audio stream's unbind never removes an entry for SSRC 0. Own edge case: unbinding an emptyStreamInfo{}and a never-bound{999, 998}left all four entries of two bound streams. - Unbind twice: idempotent, the other stream stays. Re-bind of the same SSRC after an unbind adds both entries again.
- RTX alias of stream A after stream B's main SSRC is removed: still routable (the alias stores the writer value, not a reference to B's key).
- SSRC reuse: Pion gives every new sender fresh random SSRCs (
addEncoding, rtpsender.go:200-216);AddTrackon a reused transceiver makes a new RTPSender (peerconnection.go:2489-2504). The old sender's unbind completes insideRemoveTrackbefore the room adds the next sender (signaling is serialised per room). The only way an unbind could hit another live sender's entry is a 32-bit random collision between two live senders of one connection, which was already broken before the fix (the secondAddStreamoverwrote the first). See NOTE 3.
The "unknown stream" error after an unbind
Not fatal, not a log flood. What can still write to the stream after its unbind:
- The forwarded media: no.
StopcallsReplaceTrack(nil)before the unbind loop (rtpsender.go:392-400), which calls the track'sUnbind;TrackLocalStaticRTP.Unbindtakes the track's write lock (track_local_static.go:117-131) whilewriteRTPholds the read lock across the write to every binding (track_local_static.go:195-215). So when the unbind runs, no forwarded write to that sender is in flight and none starts. Had one got the error anyway:forwardlogs a write error at most once per published track (isLoggedWriteError, room_attach.go:342-347, a CompareAndSwap) and the layer path logs at Debug (viewer_track.go:319-321). - A NACK retransmission racing the stop (chain order is bwe first, our nackResponder later, so there
is a microsecond window):
resendOneignores a write error silently (nack_responder.go:321-324). - Nothing upstream closes the connection on a write error.
Race and the forward path
passPacer.Write is unchanged (bwe.go:173-181): read lock, lookup, unlock, write. remove takes the
write lock for two map deletes. Writes of other streams are not blocked beyond that. The author's
TestPassPacerWriteRacingUnbind checks that every packet is either sent or refused and that none begun
after the unbind returned is sent. make bench-allocs: 0 allocs/op on all four BenchmarkForward cases
(base 0). The ns/op column is not judged by the script; it was taken at a load of about 7 to 11 on a
shared machine, and Write did not change.
Tests
- TestPassPacerForgetsUnboundStreams (two entries per bound stream, 0 after 500 cycles), TestPassPacerUnbindKeepsOtherStreams and TestPassPacerWriteRacingUnbind run as normal tests (the inverse-assertion known failure is gone, not skipped).
- Revert check: bwe.go reverted to 2e28b6f6c (plus a test-only shim for the missing
remove, which the tests do not call directly): all three fail ("1000 pacer entries survive 500 bind/unbind cycles", "a packet of the unbound stream was sent", "7995 packets begun after the unbind returned were sent"), and so does my real-PC probe.
Gates
No allowlist entry, limit or skipped test. bwe.go is 185 lines, the test file 142. gofmt clean, go vet
with and without -tags=integration clean, make complexity-check exit 0, node scripts/check-english-only.mjs passes, internal/repo tests pass. Commit type fix, no !, no AI
attribution. No public API change (all unexported).
NOTEs
- doc/architecture.md:479 still says "The cost is the estimator's history of 250 packets and two small goroutines per connection", while the code comment (bwe.go:21-23) now adds "one writer per stream it sends, which is forgotten when the stream's sender stops". Not wrong after the fix (per-stream cost is now bounded by live senders), but the two descriptions differ; adding the same clause to the doc would keep them in step.
- A retransmission that slips into the window between the bwe unbind and the nackResponder unbind
passes the transport-cc stamp (outer) before bwe refuses it, so it consumes one transport-wide
sequence number that never leaves. The comment in sfu_setup.go:203-205 tries to avoid exactly
such gaps for refused packets. Before the fix that packet was sent to the stopping sender instead.
Effect: at most a handful of sequence numbers per sender stop, and gcc has no send record for
them (
OnSentis in the closure that was not reached). Negligible; no action needed. - Two live senders of one connection with colliding random SSRCs (or an RTX SSRC equal to another stream's SSRC) would now lose routing for the survivor when the other stops. Before the fix the collision already misrouted packets (AddStream overwrite). Probability about n^2/2^33 per connection; not worth code.
What I ran
- Probe (temporary test file, removed): real-PC subscription toggles and publisher removal with
-race; edge cases (unrelated unbind, empty info, double unbind, rebind, alias survival). go test -race -count=1 ./internal/sfu/twice (full package: ok, 74.5 s and 75.0 s), with the probe present;-race -count=3 -run PassPacerwithout it.- Revert check as described.
make bench-allocs, gofmt, go vet (both tag sets),make complexity-check, English-only check, internal/repo tests. - All with
nice -n 10,-p 2, one heavy job at a time.
Not checked
- Heap bytes released per stopped sender (no heap profile; the leak and its release were checked by entry count and key sets only).
- Browser clients (only Pion loopback clients), camera simulcast publishers (the viewer side has one
encoding per sender regardless), and the hub's subscription message path above
Room.Subscribe. - Pion's own interceptors' unbind correctness beyond reading that the methods exist.