Review: each lower screen layer gets its own copy of the capture, and the camera's clone gaps are closed (PR #836, heads 577df35b0 and dd4c92fad)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: an agent that did not write the change. Read-only, with a focus on media and privacy. Two rounds, each in its own detached worktree, both now removed.

Final verdict

MERGE once the CI run on dd4c92fad is green. At review time no check on that head had finished.

  • The round 1 required finding is fixed and tested. A quality change now gives the lower lines new clones, which carry the new hint and constraints. The new test fails when the fix is removed (round 2, section R2.1).
  • Notes 1, 3, 6 and 7 are addressed. One cosmetic point from #824 remains (round 2, section R2.2).
  • Nothing new broke. The peer.ts change keeps the reconnect behaviour and drops the per-renegotiation syncs. E2EE is untouched (round 2, section R2.3).
  • Nothing blocks. Round 2 has four notes.

Round 1 (head 577df35b0)

Reviewer: an agent that did not write the change. Read-only, with a focus on media and privacy. I used my own detached worktree at 577df35b0, now removed. Commit reviewed: 577df35b0 (one commit). Files: web/src/talk/composables/webrtc/ layers.ts, screen.ts, peer.ts, state.ts, cameraLayers.ts, cameraLayerSync.ts, the new layerClones.ts, and three test files. No file under web/src/lib/media/ (E2EE) changed. I made mutations and one throwaway probe test in the scratch worktree only, and reverted each one after its run. Nothing was committed or pushed.

Verdict

MERGE AFTER FIXES. One fix is required, and it is small:

  1. The lower screen lines keep a stale contentHint after an "Optimise for" change. applyScreenQuality sets the new hint on the capture (senders.ts:82) and then syncLayers runs (watchers.ts:72). But layerClone reuses the held clone because the source is the same and still live (layerClones.ts:16). It copies the hint only when it makes a clone (:19). A probe test confirmed it. After a share starts as detail and switches to { resolution: 1080, fps: 30, optimize: 'motion' }, the capture reports motion, and the low line still carries the same clone with detail. Meanwhile degradationPreference on that sender is already balanced. On main, the lower lines carried the capture itself and followed the hint, so this is a regression this PR introduces, and it contradicts the PR's own "contentHint kept" claim. The fix: in layerClone, set held.clone.contentHint = source.contentHint on reuse, or stop the screen clones inside the applyScreenQuality chain step. Either needs a test that pins the hint after a quality change.

Everything else holds:

  • No path leaves a screen clone live after the share stops, and none can make a clone after the stop (section 1).
  • E2EE is unchanged (section 3).
  • Lint and the talk tests pass. The core mutations are caught (section 5).

The notes below do not block.

1. Clone lifecycle (screen)

The clones live in st.screenLayerClones (state.ts:67, :167). layerClone (layerClones.ts:14-22) makes them and dropLayerClone (:25-28) stops them. layerClone reuses a clone only while it copies the same capture and is live. Otherwise it stops the old clone first.

PathWhat happensEvidence
stopScreenShare (button, menu, logout, gaming action)stopScreenLayerClones runs synchronously, before the capture tracks are stopped. syncLayers later takes the lines offscreen.ts:239, :245, :254; layers.ts:153-156
Capture onended (the browser's own "Stop sharing")calls stopScreenShare while the stream is currentscreen.ts:35
Leaving the call or being taken overendCall runs cleanupPeerConnection (which keeps the clones) and then stopScreenShare, which stops themsession.ts:217, :219
Call becomes unprotectedonMediaUnprotected runs stopScreenShareuseWebRTC.ts:48-49
Source switched (replaceScreenShare)old clones are stopped synchronously, right before the new stream is set, with no await in between. The new clones come from the next syncscreen.ts:220-223
retuneForCapture (the capture changed size or rate)all clones are stopped synchronously. The queued sync makes new oneslayers.ts:166-170
Line no longer fed (no plan for it)dropTrack stops its clone and takes the line offlayers.ts:56-59, :122-124
No track or no best senderreleaseLayers drops every line's clone, then all clones againlayers.ts:62-71
replaceTrack failsthe clone stays held but is on no sender. The next sync reuses it, and any stop path stops it. The capture is live then anywaylayers.ts:91
Reconnect to an offer with fewer or no layer linesrunSyncLayers first drops the clones of every slot without a sender (dropMissingSenderClones), before its early return. finishOfferNegotiation now runs the sync when a share was bound or clones existlayers.ts:135-136, peer.ts:241-243

Race with the async qualityChain. feedLayer checks the current capture (layers.ts:89) and makes the clone (:90) with no await in between. Every path that ends or swaps the capture does its clone stop and its localScreenStream change in one synchronous block:

  • stopScreenShare: screen.ts:239 and :246.
  • replaceScreenShare: :220-221.
  • endCall: through stopScreenShare.

So an in-flight sync either made its clone before the stop, and the stop catches it, or it sees the new or empty stream and returns false. A clone made before the stop can still reach a sender through its pending replaceTrack, but by then it has already ended. A stopped track keeps no capture open. No clone can be created after the share stopped. The camera has the same structure (cameraLayerSync.ts:19-20).

Camera gaps from the #824 review.

  • Note 2 (unused clones after a reconnect without layer lines) is closed. dropMissingSenderClones runs before the early return (cameraLayerSync.ts:51-52). The sync is now triggered after such a reconnect (peer.ts:244). Mutations M3b and M8 catch the loss of either part.
  • Note 1 (a line no longer fed keeps its clone) is closed as well. A test now pins it (useWebRTC.cameraLayers.test.ts, "stops a lower layer's clone when the capture becomes too small"). Mutation M6 is caught.
  • The #824 property "a reconnect reuses the same clones" still holds. The drop only runs for slots without a sender, and the sync runs after the rebind. The existing test still passes.

2. Side effects

  • retuneForCapture stops the clones before the re-sync. The lower senders then sit on ended tracks until the queued chain reaches feedLayer. That wait covers anything already queued, plus one setParameters on the best line and one per lower line.
    • In the normal case this is milliseconds. The capture's size just changed, so the encoders reconfigure anyway.
    • The SFU moves an automatic viewer only after adaptStall (2 s, internal/sfu/adapt.go:61, viewer_track.go:217-229). A gap that long would need a slow applyConstraints queued ahead.
    • Stopping each clone inside the chain, right before its line is fed again, would remove the gap (note 2).
  • The changed condition in finishOfferNegotiation (peer.ts:241-246).
    • bound.screen and bound.camera are true only when the best line was bound fresh on this connection and a share or camera runs (senders.ts:137-145). That is the reconnect case, and it is cheap: with no layer lines, runSyncLayers returns after the drop.
    • The *LayerClones.some(Boolean) terms are true during every share or camera with layer lines. So every server renegotiation of the same connection (joins, leaves) now runs a full screen sync and a full camera sync. That is up to six setParameters calls, because tuneSender does not skip unchanged values (senderTuning.ts:72-81).
    • Nothing behaves wrongly: the values are the same, and declareLayers deduplicates (layers.ts:100). But it is needless work on every renegotiation.
    • The only case those terms add beyond bound.* and fresh.* is a reconnect whose best line could not be bound. Keying the sync on "this offer bound any sender fresh" would cover that without the per-renegotiation cost (note 3).
  • The best line is unchanged. It still carries the capture itself (layers.ts has no clone at index 0, and the tests pin screen.track === track).

3. E2EE

  • The lower screen senders come only from bindTransceiver(conn, mid, null, SOURCE_SCREEN, i + 1) (senders.ts:148-153). That call runs mediaCall().protectSender before it switches the line to sendrecv, and returns null when protection fails (:94-104).
  • st.layerSenders is set only from that return value and reset to null with the connection (peer.ts:30).
  • Clones are placed only in feedLayer (layers.ts:91), onto entries of a copy of st.layerSenders (:139). So every clone goes onto a sender whose transform was armed when it was bound.
  • replaceTrack swaps the track and leaves sender.transform as it is. The source (screen) and the layer index are the same as before.
  • An in-flight sync that still holds the senders of a connection closed by a reconnect puts a clone, at most, on a protected sender of a closed connection. Nothing goes out there.
  • Conclusion: the lower screen senders are protected exactly as before, and no path puts a clone on an unprotected sender.

4. contentHint and later constraints

  • At creation: layerClone copies source.contentHint (layerClones.ts:19), and a new test pins it.
  • Hint changed later: not followed. This is the required fix.
  • Constraints applied later (applyScreenQuality: track.applyConstraints(trackConstraints(quality)), senders.ts:76): a clone keeps the constraints it was cloned with.
    • The clones are recreated only through retuneForCapture. That runs when the capture watch sees new width/height/frameRate settings, after up to CAPTURE_POLL_MS + CAPTURE_SETTLE_MS (1.5 s).
    • A quality change that changes the delivered size or rate therefore converges after about 1.5 s. Until then, the lower plans are computed from the capture's settings while a clone may still deliver its old size.
    • The comment at layers.ts:164-165 gives constraints as the reason for the stop in retuneForCapture. Yet applyScreenQuality, the only place that applies constraints, does not recreate the clones itself (note 1).
  • Pre-existing, now wider: cleanupPeerConnection stops the capture watch (peer.ts:17), and the reconnect path (startWaitingScreen, peer.ts:183-192) does not start it again. Only sendScreenVideo does (screen.ts:76).
    • After a reconnect, neither the best line nor, now, the clones follow a resize or a quality change until the share is restarted.
    • This is not introduced here. With clones, though, it also leaves the lower lines on stale constraints (note 4).

5. Tests

nice -n 10 npx vitest run src/talk: 92 files, 1046 tests passed. nice -n 10 npm run lint: 0 errors, 23 warnings, none in the changed files (ESLint on the changed files alone: clean). i18n strict: 0 findings.

I ran each mutation against src/talk/composables (467 tests) and reverted it:

#MutationResult
M1Lower screen lines share the capture again (const clone = track, layers.ts:90)9 fail (own clone, reuse, stop, retune, 4K layers, announced lines, stop-and-redeclare, quality change)
M2stopScreenShare no longer stops clones (screen.ts:239 removed)1 fails (synchronous stop)
M3aScreen reconnect drop removed (layers.ts:135)1 fails (layers.test.ts, "stops clones of lines that have no sender")
M3bCamera reconnect drop removed (cameraLayerSync.ts:51)1 fails (camera reconnect test)
M4retuneForCapture no longer stops clones (layers.ts:166)1 fails
M5replaceScreenShare no longer stops clones (screen.ts:220)all pass (note 5)
M6Camera line no longer fed keeps its clone (cameraLayerSync.ts:39)1 fails
M7Screen line no longer fed keeps its clone (layers.ts:57)all pass (note 5)
M8peer.ts camera condition back to fresh.cameraLayers only1 fails
M9peer.ts screen condition back to fresh.layers onlyall pass (note 5)
M10The second clone stop in releaseLayers removed (layers.ts:68)all pass (note 6)

Probe (throwaway test, deleted): share as detail, switch to optimize: 'motion'. The capture reports motion, and the low line holds the same clone, which reports detail. That is the required fix.

6. Code quality (AGENTS.md)

  • Sizes: layers.ts 171, screen.ts 285, peer.ts 312, state.ts 218, cameraLayers.ts 55, cameraLayerSync.ts 75 and layerClones.ts 35 lines (limit 400). The tests are 781 and 259 lines (limit 800; useWebRTC.layers.test.ts is close).
  • Functions and complexity: ESLint (the function size and sonarjs cognitive complexity rules) is clean on every changed file. Extracting syncCameraSenderLayers keeps runSyncCameraLayers small.
  • Sharing: moving the clone helpers into layerClones.ts for both camera and screen is the right split. releaseCameraLayers (cameraLayers.ts:54) and stopScreenLayerClones (layers.ts:153-156) are the same loop. An exported dropAllLayerClones(slots) in layerClones.ts would remove the duplicate, and stopScreenLayerClones takes cx only to reach st.
  • Stale comments:
    • state.ts:12 still says "A lower camera layer's own copy", although the type now serves the screen too.
    • state.ts:71 still points to cameraLayers.layerClone, which moved to layerClones.ts.
    • LayerClone still sits between two import blocks (#824 note 3).
  • No dead code beyond the redundant second stop in releaseLayers (note 6).

Runs

WhatCommand / sourceResult
Setupnpm ci in web/ok
Unit, talk featurenice -n 10 npx vitest run src/talk92 files, 1046 tests passed
Web lintnice -n 10 npm run lint0 errors, 23 warnings (none in changed files); i18n strict 0 findings
ESLint, changed filesnpx eslint src/talk/composables/webrtc/ <changed tests>clean
Mutationssection 58 of 11 caught; M5, M7, M9 and M10 survive (note 5, note 6)
Probethrowaway test on the contentHint after an optimise changestale hint confirmed
CI on the headPR checks at review timesecret-scan, govulncheck, analyze (actions) green; frontend, e2e, backend and CodeQL queued

All local commands ran with nice -n 10. The load stayed under 8.

Notes (none blocking)

  1. Constraints reach the clones only through the capture watch. After applyScreenQuality, the clones keep their old constraints until retuneForCapture fires (up to 1.5 s, and only if the settings change). Stopping the screen clones in the applyScreenQuality chain step also fixes the required item. The comment at layers.ts:164-165 should then name the right trigger.
  2. Freeze window in retuneForCapture. The clones are stopped before the queued re-sync, so the lower lines carry ended tracks until the chain reaches them. This is short in practice, and well below the SFU's 2 s stall switch. Stopping each clone inside the chain, right before its line is fed, would close the gap.
  3. Needless sync on every renegotiation. screenLayerClones.some(Boolean) and cameraLayerClones.some(Boolean) (peer.ts:241, :244) run both full syncs on every server renegotiation during a share or camera, with setParameters on unchanged values. A condition on "this offer bound a sender fresh" covers the reconnect cases.
  4. Pre-existing: the capture watch is not restarted after a reconnect (peer.ts:17, screen.ts:76). It predates this PR, but now the clones also keep stale settings after a reconnect. Worth a follow-up.
  5. Test gaps.
    • M5: the replace test passes without the explicit stop, because layerClone swaps clones lazily. The test does not check that the old clones are stopped before the new capture is fed, nor the case where the sync after a replace aborts.
    • M7: no screen test checks that a line that loses its plan stops its clone. The capture runs anyway, so this is not an indicator leak.
    • M9: the screen side of the peer.ts condition is only unit-tested in runSyncLayers (layers.test.ts), not through a reconnect with fewer layer lines.
  6. Redundant stop. releaseLayers stops each line's clone in dropTrack and then all clones again (layers.ts:64-68). Together with dropMissingSenderClones (:135), the second loop is never needed.
  7. Housekeeping: the stale comments at state.ts:12 and :71, and the duplicate stop-all loop (section 6).

Not checked

  • I did not run e2e or call specs locally (rule: CI only, no camera, microphone or audio on this machine). Most CI checks on 577df35b0 were still queued at review time.
  • The browser's adaptation behaviour, and whether applyConstraints on a display capture also reconfigures its clones, rest on knowledge of Chromium's track adapters, not on a source read or a measurement. Firefox and WebKit may differ.
  • I did not measure CPU, power or memory with two extra screen clones.
  • The SFU and the E2EE code were read only where they bear on this change (stall detection, protectSender).

Outcome (round 1)

Merge after the contentHint fix and a test for it, once CI is green. Notes 1 to 3 and 5 to 7 are a small follow-up and can go into the same PR. Note 4 predates this change.


Round 2 (head dd4c92fad)

Commit reviewed: dd4c92fad, one fix commit on top of 577df35b0. It touches layerClones.ts, layers.ts, peer.ts, watchers.ts, cameraLayers.ts, state.ts and useWebRTC.layers.test.ts. No file under web/src/lib/media/ changed, and sender binding (senders.ts) is unchanged. I reviewed it in a fresh detached worktree, and made mutations there only, each reverted after its run.

Verdict (round 2)

MERGE once CI is green on dd4c92fad. Nothing blocks.

R2.1 The required finding

The fix has two parts.

  • A new clone after every quality change. The voice.screenQuality watcher now runs stopScreenLayerClones once applyScreenQuality's chain step has settled, and then syncLayers (watchers.ts:73).
    • The new clones are made after applyConstraints and after the capture got its new hint. So they copy both, which also settles round 1 note 1.
    • The .then callback is registered on the chain promise before anything queued later, so it runs before later chain steps start. A layer sync queued in between finds the clones stopped and makes new ones. The watcher's own sync then reuses them.
    • When the share was stopped or replaced in the meantime, the stop hits no clones, or the replacement's fresh ones, which the queued sync makes again right away. Nothing stays live.
  • The hint is refreshed on reuse. A reused clone now takes the capture's current hint (layerClones.ts:16-20).

Test. "gives the lower lines new copies with the new hint when the share switches to motion" (useWebRTC.layers.test.ts:699-716). It checks that the old clones are stopped, that both lower lines carry motion, and that no line still holds an old clone.

MutationResult
R1: watcher stop removed (hint refresh kept)1 fails (the new test)
R2: hint refresh on reuse removed (watcher stop kept)all pass (note R2-1)
R3: both removed, i.e. round 1's code1 fails (the new test)

Conclusion: fixed, and the test fails without the fix.

R2.2 Round 1 notes 1, 3, 6 and 7

NoteStatusEvidence
1: constraints reach the clones only through the capture watchFixed: a quality change makes new clones after applyConstraints. The comment in retuneForCapture now gives the right reasonwatchers.ts:72-73, layers.ts:159-163
3: both syncs on every renegotiationFixed: finishOfferNegotiation stops the clones of missing lines synchronously and syncs only when layer lines were bound fresh, as on mainpeer.ts:242-251
6: redundant second stop in releaseLayersFixed: removed. dropTrack covers lines with a sender, and dropMissingSenderClones at the start of runSyncLayers covers the restlayers.ts:61-70, :134
7: stale comments, duplicate stop-all loopFixed: dropAllClones in layerClones.ts serves camera and screen, and the state.ts comments are corrected. LayerClone still sits between two import blocks (cosmetic, #824 note 3)layerClones.ts:41-44, cameraLayers.ts:54, layers.ts:152-154, state.ts:12-13, :67, :71

Not in scope for this round and unchanged: round 1 notes 2 (freeze window), 4 (capture watch not restarted after a reconnect, which predates this PR) and 5 (test gaps M5 and M7).

R2.3 Does anything new break?

peer.ts (finishOfferNegotiation).

  • Reconnect with the same lines. bindPublishSenders binds the new senders before finishOfferNegotiation runs. So dropMissingSenderClones sees bound senders and keeps their clones, and the sync on the freshly bound lines reuses them. The camera reuse test from #824 still passes.
  • Reconnect with fewer or no layer lines. The clones of the missing lines are stopped right after the offer, with no sync needed. For the camera this is pinned (mutation R5 fails). For the screen it is not (note R2-2).
  • Renegotiation on the same connection. No sync and no setParameters calls any more; only two cheap synchronous loops. This is the same as main.
  • Dropped bound.screen / bound.camera terms. A share or camera that is rebound after a reconnect gets fresh layer lines whenever the server offers them, so fresh.layers / fresh.cameraLayers trigger the sync as before. Without layer lines, there is nothing to sync.
  • Early returns. If the offer turns stale (isCurrent) or the screen start fails, the drop is skipped. The next offer, or the next sync (layers.ts:134, cameraLayerSync.ts:51), catches it. The capture is live in that window, so the sharing indicator stays truthful.

Clone lifecycle still holds. The synchronous stops in stopScreenShare, replaceScreenShare and retuneForCapture are unchanged (screen.ts:220, :239; layers.ts:163). The check before layerClone still runs with no await in between (layers.ts:88-89).

MutationResult
R4: peer.ts screen drop removedall pass (note R2-2)
R5: peer.ts camera drop removed1 fails (camera reconnect)
R6: every missing-line drop removed (peer.ts and both syncs)2 fail
R7: lower screen lines share the capture again10 fail
R8: stopScreenShare no longer stops clones1 fails

E2EE: unchanged. The lower senders are still bound only through bindTransceiver with protectSender, and no new path calls replaceTrack.

Size and lint: layers.ts 168, peer.ts 316, watchers.ts 102 and layerClones.ts 44 lines. useWebRTC.layers.test.ts is at 800 lines, the exact limit, and lint accepts it (note R2-3). ESLint is clean on webrtc/.

Runs (round 2)

WhatCommand / sourceResult
Setupnpm ci in web/ok
Unit, talk featurenice -n 10 npx vitest run src/talk92 files, 1047 tests passed
Web lintnice -n 10 npm run lint0 errors, 23 warnings (none in changed files)
ESLint, webrtc/npx eslint src/talk/composables/webrtcclean
MutationsR1-R8 above6 of 8 caught; R2 and R4 survive
CI on the headPR checks at review timenone finished yet

All local commands ran with nice -n 10. The load stayed under 5.

Notes (round 2, none blocking)

  • R2-1. The hint refresh on reuse has no observable effect. The hint changes only in applyScreenQuality (senders.ts:82) and at the start of a share. Every quality change now makes new clones, so the reuse branch never meets a changed hint. Mutation R2 confirms it: all tests pass without it. Keep it as a cheap guard, with its comment saying so, or remove it as dead code (AGENTS.md: no dead code).
  • R2-2. The screen side of the missing-line drop in peer.ts is not tested. Removing peer.ts:243 leaves every test green; it is the same gap as round 1's M9. Without it, after a reconnect to an offer without screen layer lines, the old screen clones stay live until the share stops. The capture is live then, so this is not an indicator leak. A reconnect test like the camera one would pin it.
  • R2-3. useWebRTC.layers.test.ts is at exactly 800 lines. The next screen-layer test has to go into a split file.
  • R2-4. A short freeze now also happens on every quality change. It is the same class as round 1 note 2: the lower lines carry ended tracks from the watcher's stop until the queued sync feeds them. The quality change reconfigures the encoders anyway, and the gap is far below the SFU's 2 s stall switch.

Not checked (round 2)

  • I did not run e2e or call specs locally (CI only, no camera, microphone or audio on this machine). CI on dd4c92fad had not finished at review time.
  • I did not verify in a real browser that a clone made after applyConstraints carries the capture's new constraints (Chromium behaviour, from knowledge, not measured).

Outcome (round 2)

Merge once CI on dd4c92fad is green. Notes R2-1 and R2-2 are a small follow-up.

Something wrong on this page? Tell us.