Fix review: H-L8-03 (bandwidth estimator pacer never forgets a stopped stream)

From mnema-talk v3.3.13, rendered on this site.

Fix: branch msch128/fix-h-l8-03, commit ef94fb6db ("fix: forget a stopped stream in the bandwidth estimator's pacer"), on the lane 8 test commit 2e28b6f6c. Reviewed: the diff of ef94fb6db only (internal/sfu/bwe.go +25/-2, internal/sfu/bwe_writers_leak_test.go). Read alongside: the finding, the repro and refute reports, internal/sfu/{bwe.go, sfu_setup.go, room_signal.go, room_attach.go, viewer_track.go, nack_responder.go}, pion/webrtc v4.2.26 (rtpsender.go, peerconnection.go, rtptransceiver.go, track_local_static.go) and pion/interceptor v0.1.51 (chain.go, pkg/gcc, internal/cc, pkg/twcc, pkg/report, pkg/stats, pkg/nack). Temp git worktree, removed afterwards.

Verdict: MERGE

No BLOCKER, no SHOULD-FIX. Three NOTEs.

Does it remove the cause

Yes. bweInterceptor.UnbindLocalStream (bwe.go:101-103) calls passPacer.remove (bwe.go:164-171), which deletes info.SSRC and, only when non-zero, info.SSRCRetransmission under the pacer's write lock. This mirrors BindLocalStream exactly (bwe.go:88-96: est.AddStream registers info.SSRC through gcc, the RTX alias is added only when SSRCRetransmission != 0). FEC SSRCs were never registered, so there is nothing to forget for them.

Bind and unbind receive the same values in production. Pion binds once per encoding in RTPSender.Send with &trackEncoding.streamInfo (rtpsender.go:350-366) and unbinds the same pointer in RTPSender.Stop (rtpsender.go:400). The SFU stops viewer senders only through PC.RemoveTrack (room_signal.go:181, 225), which calls sender.Stop() (peerconnection.go:2536); PeerConnection.Close stops every sender the same way. The interceptor chain calls every interceptor's UnbindLocalStream (chain.go:47-51), so ours is reached.

Simulcast: viewer senders are created only by PC.AddTrack (room_signal.go:215) with one track, so each has exactly one encoding and one bind. Layer groups give the viewer one track and switch the source inside it (layer_viewers.go), so a layer switch is no bind and no unbind. Even a multi-encoding sender would be symmetric: Send binds and Stop unbinds once per encoding, each with its own StreamInfo.

gcc: SendSideBWE.AddStream (send_side_bwe.go:153-180) wraps the writer in a closure that is stored only in our pacer (e.pacer.AddStream) and returns the pacer itself. SendSideBWE holds no stream map (its fields: pacer, loss and delay controllers, feedback adapter, bitrate state); the feedback adapter's history is a fixed 250-entry list of (SSRC, sequence) send records with no writer reference. After the delete nothing in gcc references the stopped sender's writer chain, so gcc can neither call it nor keep it alive. The other registered interceptors: the transport-cc stamp keeps a single counter (no per-stream state); Pion's report sender, stats and NACK responder implement UnbindLocalStream; the NACK generator is a remote-stream interceptor with UnbindRemoteStream; ours (nackResponder) unbinds its stream (nack_responder.go:134-143). The commit message's inventory is correct.

Through a real Pion PeerConnection (the lead's extra point)

Own temporary probe test (not committed), using the package's loopback clients and the SFU's real registry (NewSFU -> buildAPI -> buildInterceptors). It wraps newEstimator to capture each connection's *gcc.SendSideBWE and reaches its pacer through AddStream (which returns the pacer), removing the sentinel it added. Publisher: screen (VP8, with RTX) and microphone (Opus, no RTX). Viewer: joins, then 15 times Room.Subscribe(screen, on) and (screen, off), waiting each time for the renegotiation; then RemovePeer of the publisher.

After every step the set of keys in the viewer's passPacer.writers was compared with the set {encoding SSRC, RTX SSRC} of the viewer's live senders (GetSenders()[i].GetParameters()):

initial:               pacer [mic]                senders [mic]
screen on:             pacer [mic, scr, scr-rtx]  senders [mic, scr, scr-rtx]
after 15 on/off:       pacer [mic]                senders [mic]
after publisher left:  pacer []

The sets were equal after every step, so the unbind gets the same SSRC and RTX SSRC as the bind, and stopping the screen sender never removed the microphone's entry. The captured SFU log had zero "unknown stream" lines. With bwe.go reverted to 2e28b6f6c the same probe times out at the first "screen off" (the screen's two entries stay).

Can one sender's unbind remove another live sender's entry

  • Different SSRCs: no (own probe above, and the author's TestPassPacerUnbindKeepsOtherStreams).
  • RTX zero: remove skips it, so an audio stream's unbind never removes an entry for SSRC 0. Own edge case: unbinding an empty StreamInfo{} and a never-bound {999, 998} left all four entries of two bound streams.
  • Unbind twice: idempotent, the other stream stays. Re-bind of the same SSRC after an unbind adds both entries again.
  • RTX alias of stream A after stream B's main SSRC is removed: still routable (the alias stores the writer value, not a reference to B's key).
  • SSRC reuse: Pion gives every new sender fresh random SSRCs (addEncoding, rtpsender.go:200-216); AddTrack on a reused transceiver makes a new RTPSender (peerconnection.go:2489-2504). The old sender's unbind completes inside RemoveTrack before the room adds the next sender (signaling is serialised per room). The only way an unbind could hit another live sender's entry is a 32-bit random collision between two live senders of one connection, which was already broken before the fix (the second AddStream overwrote the first). See NOTE 3.

The "unknown stream" error after an unbind

Not fatal, not a log flood. What can still write to the stream after its unbind:

  • The forwarded media: no. Stop calls ReplaceTrack(nil) before the unbind loop (rtpsender.go:392-400), which calls the track's Unbind; TrackLocalStaticRTP.Unbind takes the track's write lock (track_local_static.go:117-131) while writeRTP holds the read lock across the write to every binding (track_local_static.go:195-215). So when the unbind runs, no forwarded write to that sender is in flight and none starts. Had one got the error anyway: forward logs a write error at most once per published track (isLoggedWriteError, room_attach.go:342-347, a CompareAndSwap) and the layer path logs at Debug (viewer_track.go:319-321).
  • A NACK retransmission racing the stop (chain order is bwe first, our nackResponder later, so there is a microsecond window): resendOne ignores a write error silently (nack_responder.go:321-324).
  • Nothing upstream closes the connection on a write error.

Race and the forward path

passPacer.Write is unchanged (bwe.go:173-181): read lock, lookup, unlock, write. remove takes the write lock for two map deletes. Writes of other streams are not blocked beyond that. The author's TestPassPacerWriteRacingUnbind checks that every packet is either sent or refused and that none begun after the unbind returned is sent. make bench-allocs: 0 allocs/op on all four BenchmarkForward cases (base 0). The ns/op column is not judged by the script; it was taken at a load of about 7 to 11 on a shared machine, and Write did not change.

Tests

  • TestPassPacerForgetsUnboundStreams (two entries per bound stream, 0 after 500 cycles), TestPassPacerUnbindKeepsOtherStreams and TestPassPacerWriteRacingUnbind run as normal tests (the inverse-assertion known failure is gone, not skipped).
  • Revert check: bwe.go reverted to 2e28b6f6c (plus a test-only shim for the missing remove, which the tests do not call directly): all three fail ("1000 pacer entries survive 500 bind/unbind cycles", "a packet of the unbound stream was sent", "7995 packets begun after the unbind returned were sent"), and so does my real-PC probe.

Gates

No allowlist entry, limit or skipped test. bwe.go is 185 lines, the test file 142. gofmt clean, go vet with and without -tags=integration clean, make complexity-check exit 0, node scripts/check-english-only.mjs passes, internal/repo tests pass. Commit type fix, no !, no AI attribution. No public API change (all unexported).

NOTEs

  1. doc/architecture.md:479 still says "The cost is the estimator's history of 250 packets and two small goroutines per connection", while the code comment (bwe.go:21-23) now adds "one writer per stream it sends, which is forgotten when the stream's sender stops". Not wrong after the fix (per-stream cost is now bounded by live senders), but the two descriptions differ; adding the same clause to the doc would keep them in step.
  2. A retransmission that slips into the window between the bwe unbind and the nackResponder unbind passes the transport-cc stamp (outer) before bwe refuses it, so it consumes one transport-wide sequence number that never leaves. The comment in sfu_setup.go:203-205 tries to avoid exactly such gaps for refused packets. Before the fix that packet was sent to the stopping sender instead. Effect: at most a handful of sequence numbers per sender stop, and gcc has no send record for them (OnSent is in the closure that was not reached). Negligible; no action needed.
  3. Two live senders of one connection with colliding random SSRCs (or an RTX SSRC equal to another stream's SSRC) would now lose routing for the survivor when the other stops. Before the fix the collision already misrouted packets (AddStream overwrite). Probability about n^2/2^33 per connection; not worth code.

What I ran

  • Probe (temporary test file, removed): real-PC subscription toggles and publisher removal with -race; edge cases (unrelated unbind, empty info, double unbind, rebind, alias survival).
  • go test -race -count=1 ./internal/sfu/ twice (full package: ok, 74.5 s and 75.0 s), with the probe present; -race -count=3 -run PassPacer without it.
  • Revert check as described. make bench-allocs, gofmt, go vet (both tag sets), make complexity-check, English-only check, internal/repo tests.
  • All with nice -n 10, -p 2, one heavy job at a time.

Not checked

  • Heap bytes released per stopped sender (no heap profile; the leak and its release were checked by entry count and key sets only).
  • Browser clients (only Pion loopback clients), camera simulcast publishers (the viewer side has one encoding per sender regardless), and the hub's subscription message path above Room.Subscribe.
  • Pion's own interceptors' unbind correctness beyond reading that the methods exist.

Something wrong on this page? Tell us.