Review: each lower camera layer gets its own copy of the capture, and the e2e noise filter is turned off from the fixture (PR #824, head 90ef9d169)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: an agent that did not write the change. Read-only, with a focus on media and privacy. I used my own detached worktree at 90ef9d169, which sits on current main and is now removed. Commits reviewed: 26b094fe3 (one clone of the capture per lower camera layer) and 90ef9d169 (trackPeerConnections added from the fixture to every e2e context). The same two patches were tested in CI as c880351e4 and 225f30a8f. git range-diff shows them identical, rebased onto a newer main. CODEOWNERS paths touched: none that hold crypto. The client change is in web/src/talk/composables/webrtc/, and the E2EE code (web/src/lib/media/) is unchanged. I made mutations in the scratch worktree only and reverted each one after its run. Nothing was committed or pushed.

Verdict

MERGE once the CI run on 90ef9d169 is green. At review time it was queued: CI, CI (macOS calls) and CodeQL. The identical patches at 225f30a8f passed all three.

  • The root-cause claim fits Chromium's adaptation model and the CI numbers exactly (section 1).
  • I found no path where a clone outlives the camera. Every way the camera stops goes through releaseCameraLayers, which stops every clone (section 2).
  • E2EE is unchanged. Clones only ever reach senders whose transform was armed when they were bound, and a track swap does not touch sender.transform (section 3).
  • The unit tests fail when the lines share the track again (section 5).
  • The fixture change removes no assertion, and no spec still records connections twice (section 6).

There are no blocking findings. There are six notes, below.

1. Is the stated cause the real one?

Mechanism. In Chromium, each web MediaStreamTrack that goes to a sender gets one WebRTC video track with its own source. That source adapts frames to the combined sink wants of every encoder it feeds, and the most restrictive max_pixel_count wins.

On main, the best line and both lower lines were three separate senders on the same track. When the best line's encoder asked for fewer pixels because of bandwidth (degradationPreference: 'balanced', senders.ts:43), the shared source shrank its frames for all three lines. Each lower line then applied its fixed scaleResolutionDownBy on top of that.

A clone is a new web track on the same capture source, so it gets its own adapting source. This is the same reason simulcast layers inside one sender do not drag each other down.

CI evidence. I read run 43503313142, job e2e-shard (1/4) (id 132156049825). Its head was 74f978e9b: the fixture commit plus a diagnostic, without the client fix. The [sharer] lines at log lines 1762-1764 show:

MomentBest line (mid 2)640 line (mid 6)Thumbnail line (mid 7)
tile1280x720, limit=none, inactiveinactive320x180, limit=none, changes=0
stage640x360, limit=bandwidthinactive160x90, limit=none, changes=0
back to tile (45 s later)inactiveinactive240x135, limit=none, changes=0
  • 160x90 is 640x360 / 4, and 240x135 is 960x540 / 4. The thumbnail followed the best line's adapted input exactly.
  • The thumbnail's own encoder never reported a limitation or a resolution change.
  • The assertion toBe(180) failed with Received: 135 (camera-layers.spec.ts:133).

That is the pattern a shared adapting source produces. An encoder that limits itself would show limit=bandwidth or cpu and a non-zero change count.

With the fix. On 225f30a8f, job e2e-shard (1/4) (132189945127) passed camera-layers.spec.ts:74 on the first attempt in 57.9 s. The project has retries: 1, and no flaky result was reported. camera-visibility.spec.ts, which failed in the earlier run at its join step (line 72, unrelated to layers), also passed.

Verdict: real cause, and the fix targets it.

2. Clone lifecycle (camera light)

The clones live in st.cameraLayerClones (state.ts:70, :165). They are made in layerClone (cameraLayers.ts:53-61) and stopped in dropLayerClone (:64-67). layerClone reuses a held clone only when it copies the same capture and is still live. Otherwise it stops the old clone first.

PathWhat happensEvidence
stopCamera (button, toggleCamera)releaseCameraLayers takes every line off and stops every clone, then the capture stopscamera.ts:91, cameraLayers.ts:74-77
Capture onended (device gone)calls stopCamera, same as abovecamera.ts:38
Leaving the call, being taken overendCall runs cleanupPeerConnection and then stopCamera. The senders are already null, but the clones array is not, so the clones are still stoppedsession.ts:217, :220
Call becomes unprotectedonMediaUnprotected runs stopCamerauseWebRTC.ts:48-51
restartCamera / new devicestopCamera first, then startCamera. A leftover clone of the old capture would be replaced because source no longer matchescamera.ts:102-106, cameraLayers.ts:55-56
Line no longer fed (no plan for it, or no track)line taken off and its clone stoppedcameraLayerSync.ts:38-41
replaceTrack on a line failsthe clone stays held but is on no sender. It is stopped by the next stopCamera, or reused by the next synccameraLayerSync.ts:19-20, cameraLayers.ts:37-43
Reconnect / new connectioncleanupPeerConnection resets the senders but keeps the clones. The next syncCameraLayers puts the same clones on the new lines, and no new copy is made (test at :180)peer.ts:31, :245
Sync in flight while the camera stopsthe check before layerClone runs synchronously against st.localCameraStream. stopCamera clears that stream in the same synchronous block as releaseCameraLayers, so no clone can be made after it. A clone already passed to a pending replaceTrack has already been stopped. A stopped track keeps nothing opencameraLayerSync.ts:18-20, camera.ts:86-95
Capture discarded after a late getUserMediano clone exists yetcamera.ts:72-75

st.localCameraStream is replaced only by startCamera, which requires it to be empty (camera.ts:29). So no path swaps the capture without going through stopCamera.

No path leaks a clone beyond the camera's own lifetime. Two paths keep a clone alive but unused while the camera runs: a failed replaceTrack, and a reconnect whose offer has no camera layer lines (note 2). In both cases the main capture is live anyway, so the camera light reflects the truth.

3. E2EE

  • Media frames are sealed per sender. bindTransceiver (senders.ts:94-104) calls mediaCall().protectSender before it sets the line to sendrecv, and returns null when that fails.
  • protectSender ends in guard.arm, which sets sender.transform = new RTCRtpScriptTransform(...) with the line's source and layer (guard.ts:193-212).
  • st.cameraLayerSenders[i] is only ever set from that return value (senders.ts:155-158), and is reset to null with the connection (peer.ts:31).
  • Clones are placed only through feedLowerLayer onto those senders (cameraLayerSync.ts:19-20). So a clone can only go onto a sender whose transform is already armed.
  • replaceTrack swaps the track and leaves sender.transform and its port as they are. The transform options (source camera, layers 1 and 2) are the same as before.
  • Between a reconnect and the rebind, the held clones are on no sender at all.

Conclusion: the lower-layer senders are protected exactly as before, and no path sends a clone before its sender is protected. No file under web/src/lib/media/ changed.

4. Cost (reasoned, not measured)

  • Capture: unchanged. A clone shares the capture source, so the camera is opened and read once.
  • Encoders: unchanged. There was already one encoder per line, and inactive lines (active: false) encode nothing.
  • Per frame: each clone adds one track and one WebRTC source on the frame path. The frame buffer is passed by reference, not copied. When a line's source adapts, it may scale the frame itself. Before, one shared source did that scaling for all lines. Now up to two extra downscales can run per frame, and only while those lower lines adapt. This is small next to encoding.
  • Memory: a few objects per clone, no extra frame buffers in steady state.
  • Behaviour change: the lower lines no longer shrink with the best one. Under a bandwidth limit, the 640x360 line can now be larger than an adapted best line. Its 500 kbit/s cap still bounds what it costs. Before the fix this inversion could not happen (note 4).
  • Not measured: CPU, power and memory on a real device, with all three lines active and while adapting. CI does not measure them, and this review ran no browser.

5. Tests

useWebRTC.cameraLayers.test.ts: 10 tests pass. In the scratch worktree I mutated cameraLayerSync.ts and cameraLayers.ts, one mutation per run, and reverted each one:

MutationResult
The lines share the capture again (const clone = track)4 fail: declares-before-first-video, own-copy, refuses-the-track, new-connection
Both lower lines share one clone (layerClone(st, 0, ...))3 fail
releaseCameraLayers no longer stops the clones1 fails (own-copy, stop not called)
dropLayerClone does not call stop()1 fails (own-copy)
No reuse: a fresh clone on every sync2 fail (own-copy, new-connection)
A line no longer fed keeps its clone (cameraLayerSync.ts:40 removed)all pass (note 1)
  • The new test (:135) pins one clone per line, that the best line keeps the capture, reuse across a later sync, and the stop on stopCamera.
  • The changed assertions at :45, :162 and :205-209 pin clones in place of the capture, and no new copy on a new connection.

6. The fixture change (e2e/tests/fixtures.ts:240-244, helpers.ts:87-126)

  • Every context gets the recorder, before the spec's own scripts. counted patches browser.newContext on the worker's browser, including the WebKit contextOwnProcess path. It adds trackPeerConnections before it returns the context, so the script is registered before any script a spec adds. Playwright runs init scripts in the order they were added.
  • Other filters still win. late-join (:19), audio-stall (:22) and perf-idle (:67) set ai, and opus-dtx (:23) sets browser. Each does so unconditionally in its own init script, which runs after the recorder.
  • mnema_noise is written only while unset (helpers.ts:107-109), inside a try for frames without storage. This also settles note 2 of the #816 review: a filter picked in the settings now survives a reload.
    • No spec asserts the default filter. noise and deepfilter appear only in the four specs above, the fixture and the helper.
  • No assertion was removed. Every removed line is an import or an addInitScript(trackPeerConnections) call. The rest are a private recorder body (hidden-video, screen-layers including keepPeers, talk-extras) or a comment. The counts of expect(, expect.poll and test( are equal before and after in all 26 changed specs.
  • No spec records connections twice. The only other RTCPeerConnection wrappers are forceRelay in relay-call.spec.ts:24-31 and connection-test.spec.ts:16-23. They do not push to __pcs. They are added after the recorder, so they subclass it: one entry per connection, with the relay policy applied.
  • camera-layers.spec.ts now adds sentVideo (:57-71) and prints it only when the thumbnail step fails (:134). It only reads state and asserts nothing new.

Runs

WhatCommand / sourceResult
Unit, PRnice -n 10 npx vitest run src/talk/composables/useWebRTC.cameraLayers.test.ts10 passed
Unit, talk featurenice -n 10 npx vitest run src/talk92 files, 1038 tests passed
Mutationssection 55 of 6 caught
Web lintnpm run lint in web/0 errors, 23 warnings, none in the changed files (ESLint on them alone: clean); i18n strict 0 findings
e2e typechecknpm run typecheck in e2e/pass
CI before the fixrun 43503313142 (head 74f978e9b), job e2e-shard (1/4) 132156049825camera-layers failed, 135 for 180, stats as in section 1
CI on the identical patchesrun 43511376031 (CI), 43511375854 (macOS calls), 43511375961 (CodeQL), head 225f30a8fall success; camera-layers passed first try (57.9 s) in shard 1/4
CI on the headruns 43526394822, 43526394861, 43526394864, head 90ef9d169queued at review time

All local commands ran with nice -n 10 (load stayed under 3).

Notes (none blocking)

  1. Test gap: no test checks that a line that is no longer fed stops its clone (cameraLayerSync.ts:40). Removing that line leaves all 10 tests green. The effect is bounded: the camera runs anyway, and stopCamera stops the clone later. A case with a capture that shrinks below a layer's plan, or with a camera too small after a restart, would pin it.
  2. Unused clones after a reconnect without layer lines. runSyncCameraLayers returns early when no camera layer sender is bound (cameraLayerSync.ts:28). It also skips a null sender (:36). So after a reconnect to an offer with fewer or no camera layer lines, the old clones stay live and unused until the camera stops. This is not a camera-light leak, because the capture is live. Dropping the clones of unbound lines there would make the state match the lines.
  3. Style: LayerClone is declared between two import blocks (state.ts:12-14). Lint accepts it. Moving it below the imports would read better.
  4. Behaviour change under congestion. Each line now adapts on its own, so a bandwidth-limited best line can come out smaller than the 640x360 line. A viewer on the stage then receives less than one on the medium layer would. Simulcast avoids this by dropping top layers instead of shrinking them. The change is acceptable and capped by each line's maxBitrate, but no test or CI step watches it.
  5. The screen share has the same pattern. feedLayer puts the screen track itself on every lower screen line (layers.ts:78). With maintain-resolution (the detail preset) the encoder does not ask for fewer pixels, so the effect should be limited to the motion presets. Worth a follow-up check with the same sentVideo diagnostic. It is out of scope here.
  6. Cost not measured (section 4).

Not checked

  • I did not run e2e or call specs locally (rule: CI only, no camera, microphone or audio on this machine). The behaviour claims rest on the unit tests and on CI.
  • The CI run for head 90ef9d169 had not started at review time.
  • I did not read the Chromium source in this review. The adaptation model in section 1 is from knowledge of Chromium's WebRTC track source and sink-wants aggregation, and it is consistent with the CI numbers. Firefox and WebKit may adapt differently; the cross-browser CI jobs passed on 225f30a8f.
  • I did not measure CPU, power or memory (section 4).
  • I did not re-read the 22 smaller spec diffs line by line beyond the removed-line scan and the assertion counts.

Outcome

Merge after the CI run on 90ef9d169 passes. The notes need no change before merge. Notes 1 and 2 are a small follow-up.

Something wrong on this page? Tell us.