Review: the monotonic pacer and back-offs (H-L2-05, H-L9-02), the availability budget, and five simulator seeds
From mnema-talk v3.3.13, rendered on this site.
Branch: msch128/fix-h-l2-05-l9-02 at 9fe070ce3 (base 80a176920 lane 2's corrected sweeps; merges of hunt-lane9 and of
fix-h-l2-04; then f8e6e5667 the fix and 9fe070ce3 the regenerated failure lists). Read-only review in a temporary checkout;
my scratch tests were removed. One finding file written: findings/H-L2-06.md (in the repository: doc/security/findings/2026-11-hunt/h-l2-06.md).
Summary
- Part A (fix
f8e6e5667): MERGE. - Part B (availability regression of the merged tree): acceptable as is, with the budget table set to the merged tree's numbers (table below). The pooled hostile loss does not move (12.74 to 12.82 percent over all 400 seeds); the shard 0 breach is the reshuffling of trajectories that any protocol change causes. H-L2-04 should ship.
- Part C: seed 5 and seed 120 are one real defect (H-L2-06); seeds 20 and 82 are harness matters; seed 102 is a slow tail that H-L2-04's timing moved over the 240 s line, cause not traced.
Part A: fix review of f8e6e5667
Verdict: MERGE
What I checked
The change:
pacer.tsandbackoff.tsdefault toclock.now()through a closure (so a test that replacesclock.nowis followed); the four construction sites pass it explicitly (browserCall.ts:15,groupBase.ts:56,60,61). Nonew WirePacer(ornew Backoff(is left without it (grep over non-test code). The diff of the commit outside tests and comments is exactly those files: nothing else of the protocol changed.Remaining wall readers in non-test
lib/media:clock.ts:14(fallback when there is noperformance),clock.ts:20(clock.wall),groupEpoch.ts:106,184(lastTickWall) andgroupBase.ts:318(sleepGap, the larger of the monotonic and the wall gap). That is the one deliberate wall reader the author names; a step back cannot move it (the wall gap goes negative, the monotonic gap wins), a step forward can only make a tab look as if it slept (see seed 120 below).Revert checks (four production files restored to the parent, then each alone):
Reverted Result all four files 3 tests fail (pacer sends 5 of 10, back-off waits 3603000 ms, the room is apart for 314 and 1304 s) pacer.tsonlypacer.step.test.tsfailsbackoff.tsonlybackoff.test.tsfailsgroupBase.tsonly (the three construction sites)nothing fails browserCall.tsonly (the pacer's construction site)nothing fails The last two are not a defect: the defaults are monotonic now, so passing the clock explicitly is redundant, and the tests fail when a default turns back to the wall clock. It is the one thing the tests do not pin: a site that later builds its own clock.
Gates:
src/lib/media48 files passed, 6 skipped, 450 tests passed, 87 skipped (exit 0); ESLint onsrc/lib/mediaexit 0;vue-tsc --noEmitfortsconfig.app.jsonandtsconfig.tests.jsonexit 0.groupBase.ts344 lines,pacer.ts61,backoff.ts41.
The claim about performance.now across sleep
The commit says both behaviours are safe and I agree with the argument in the code. monotonicNow is performance.timeOrigin + performance.now(). Whether performance.now() advances while the machine sleeps depends on the browser and the OS (it
does on some platforms and does not on others, which is why the sleep check reads the wall clock too). Standing still: a wait
resumes after waking with its awake remainder, never longer than its own length (pacer one second; back-offs at most their
ceiling, 60 s for swaps and resets, 10 minutes for rivals). Jumping forward: the wait is over, as the time really passed, and
a back-off that is quiet starts from its base. A backward move cannot happen. I did not run a real browser; this is the code
and the specifications.
Findings
- NOTE 1. After a sleep during which the monotonic clock stood still, a rival back-off opened just before the sleep still holds
for its remaining awake time, up to 10 minutes (
RIVAL_MAX_MS), where the wall clock would have released it. That delays the merge of two groups after a wake by up to the ceiling; it cannot loop and is inside the constants. Acceptable. - NOTE 2. The tests do not pin the construction sites (table above). A one-line test that builds a
MediaGroupand checks that its three back-offs follow a replacedclock.nowwould. Not needed to merge. - NOTE 3. SECURITY.md already says the key schedule is measured with the monotonic clock (line 948) and does not mention the
pacer or the back-offs, so no text is wrong. The remaining wall reader (sleep check) is described in the commit message and the
clock.tsheader only; one sentence in SECURITY.md would complete it. - NOTE 4 (not about this commit). SECURITY.md in the merged tree says "A receiver never accepted a removal earlier than the limit".
The rule is
now - lastAlive <= absentMs - slack(groupVet.ts,prematureRemoval): a removal is accepted fromlimit minus one maintenance intervalof silence by the receiver's own clock (19.5 minutes in production, 28 s in the simulator). The sentence should say "never earlier than the limit minus one maintenance interval". Seed 20 below is a consequence.
What I did not check
A real browser; the heartbeat behaviour of the desktop client; whether any code outside lib/media builds a pacer or a back-off
(grep of web/src: none).
Part B: the availability budget
Method
The nightly availability sweep (SIM_AVAIL=1 SIM_SHARD=<i>/8 SIM_AVAIL_OUT=..., groupAvailability.sweep.test.ts), all 8 shards,
both relays (50 seeds each per relay per shard), at the merged tree and with H-L2-04's retry switched off (the first line of
GroupBehind.refused returns the step: the merged tree minus H-L2-04, which is lane2-b's behaviour in this code since the fix
only adds groupBehind.ts and the flag). My "without" numbers reproduce the committed shardLost table of lane 2 exactly
(0.1223, 0.1076, 0.1215, 0.1479, 0.1068, 0.1221, 0.167, 0.0984 for hostile), so the two trees are the ones compared.
Three processes of one worker each, nice -n 10, load below 12.
Pooled share of member time that lost its keys after holding them
Hostile relay, per shard:
| Shard | lane2-b behaviour (budget today) | merged tree |
|---|---|---|
| 0 | 12.23 percent | 15.05 |
| 1 | 10.76 | 10.67 |
| 2 | 12.15 | 12.95 |
| 3 | 14.79 | 13.73 |
| 4 | 10.68 | 11.14 |
| 5 | 12.21 | 11.79 |
| 6 | 16.70 | 14.80 |
| 7 | 9.84 | 11.08 |
| all 400 seeds | 12.74 | 12.82 |
Honest relay, per shard: 6.42, 4.46, 5.81, 5.17, 6.31, 6.79, 7.30, 7.14 to 6.43, 4.60, 5.81, 5.17, 5.93, 6.69, 7.38, 6.46
(all 400 seeds: 6.22 to 6.10). Start-overs: honest 2118 to 2129, hostile 5627 to 5682. Violating seeds: none on either relay in
either tree.
Worst seeds, hostile: lane2-b behaviour 34.1 (228), 30.8 (256), 29.9 (174); merged 38.9 (187), 38.2 (352), 36.3 (107).
Seed 352 goes from 9.1 to 38.2 percent. Worst longest stretch without keys: hostile 160 s to 205 s (seeds 294, 383, 352); first-keys
latency unchanged (134.5 s hostile, 97 s honest). Time to agree after the relay stops: hostile median 16 s to 19 s, 90th percentile
106 s to 86 s; 41 seeds more than 30 s slower and 43 more than 30 s faster; honest median 8 s to 8 s.
Judgement
Acceptable as is, with the table set to the merged tree's numbers.
- The effect is not systematic. Over 400 hostile seeds the paired difference of the lost-key share has mean +0.88 points and a standard deviation of 9.4 points per seed (a standard error of about 0.5), while the member-time-weighted pool is 12.74 to 12.82 percent. 230 of 400 hostile seeds change trajectory (17 of 400 on the honest relay) and they move up and down in equal measure: a shard of 50 seeds swings by about 1.3 points just from the reshuffle, so the budget's window of 2 points per shard is around 1.5 standard deviations of what any change to the group's timing does. Shard 0 (+2.8 points) and shard 6 (-1.9) are the two ends of that spread.
- The intended price. The retry rule makes a member that refused a commit for an unseen add start over after 4 s; the member without keys is then counted in the metric, where a member left behind with stale keys was not. Over the 400 hostile seeds the new path fired 2120 holds and 261 start-overs (of 5682 start-overs in all, 4.6 percent); 88 percent of the holds were resolved by the package arriving. On the honest relay 157 holds and 28 start-overs (1.3 percent). Both relays: no invariant broke; the room agrees at least as fast at the median.
- No liveness defect found in the fix. In seed 352 the new path made one start-over (m2 at 15.5 s) and 19 holds, the last at 76 s, none of which led to a start-over after 15.5 s; the long stretch
(four members without keys from 75 s to 217 s) is not a start-over of the new path.
The trajectory differs from the first start-over on, and the members' stretch comes from the chain of
leaf swaps by stale packages (
commit[remove,add]every epoch) that the hostile relay's replays cause, which lane2-b has as well on other seeds (its worst is 34 percent). Same count of start-overs in both runs (44 and 43), different timing. The retry and the start-over fire only through the flag "the first violation was an unseen add" (reviewed earlier); a hold that is not an unseen add cannot start anything (a mutant test kills that). A hostile relay cannot keep a member out for good by this path where it could not before: the start-over is bounded by the reset limit, the relay can already stop any member's package from arriving, and a member that cannot start over because of the limit stays behind exactly as before. - Set the table to the merged numbers. Hostile
shardLost:[0.1505, 0.1067, 0.1295, 0.1373, 0.1114, 0.1179, 0.148, 0.1108]; honest:[0.0643, 0.046, 0.0581, 0.0517, 0.0593, 0.0669, 0.0738, 0.0646]. The per-seed limit for a seed outside the failure budget (seed: 0.35for hostile) is below three measured seeds (38.9, 38.2, 36.3): raise it to 0.40 or the nightly will fail on seeds 187, 352 and 107. The table is new, so this is setting a budget, not lowering a gate; say in the comment that it was measured on the merged tree and that per-shard values swing by about 1.3 points with any change of timing. Optional improvement: make the nightly check the pooled value over all 8 shards (12.8 hostile, 6.1 honest, window 1 point) rather than each shard alone, which is the statistic that did not move.
I did not run lane2-b itself (a second checkout), only its behaviour in the merged code, which reproduces its committed table to four digits.
Part C: the five seeds
All traced with a scratch runner on the merged tree (runSeed with the plan named). I give the repro of the one real defect in
findings/H-L2-06.md (in the repository: doc/security/findings/2026-11-hunt/h-l2-06.md).
1. Seed 5, hostile, every plan with steps: a real defect, one finding (H-L2-06)
End state at 315 s: two groups with the same members {1,7}, both at epoch 14, different keys
(m7#4:14:... {1,7} | m1#8:14:... {1,7}). It fails with plan steps, steps-back and all, and passes with none, as the author says; with
the wall steps made no-ops it fails too (I made stepWall a no-op and ran it: still fails), so it is not the clock.
It is not permanent: with the room given 2 hours of virtual time it agrees 486 s after the relay stops (at the simulator's compressed lifetimes). At production ratios the same seed and plan settle (a different history).
Why the two groups never see each other, from the trace and the code: both groups have the same ids and the same epoch NUMBER. A
heartbeat names the group, the id, the epoch number and a time (heartbeat.ts), not the tree's state; the leaf of the lowest id in
each group is signed with the same key, so its heartbeats verify in the other group and count as a sign of life when the numbers are equal
(SECURITY.md says so: "when the two epoch numbers happen to be equal; they make nobody start over"). The detector for
another state needs the numbers to differ at two heartbeats at least 8 s apart with the member's own state unchanged in between, while
both groups rotate and renew. Welcomes do not help: m7's welcome to m1#8 is ignored because m1#8's committer is itself, and m1#8's welcome
to m7 is not from "a lower id outside my group" (its sender id is the id of m7's committer, rivalry being judged by id, not leaf).
A zombie leaf of the lowest id in the other group therefore stays alive for as long as the numbers coincide.
Belongs on the known-failure list with the finding id H-L2-06 (the list comment should say "equal-epoch fork", not "the plan's schedule"). Not a harness matter.
2. Seed 120 of plan "all", HONEST relay: the same defect; not permanent; the sleep check moves the history, it is not the cause
End state: {2,4,6} (m4, m6) and {2,4,6,11} (m2, m11), both at epoch 14, different keys: the same equal-epoch fork as seed 5, now on an honest relay.
It is not permanent: with 2 hours it agrees after 304 s (the invariant allows 240 s).
Is the sleep check the cause? I ran three variants. With the wall step made a no-op (stepWall returns before it touches the clock) the seed passes (agrees after 9 s). With the step applied but the
simulator's "woke at" mark removed, it still fails. With sleepGap ignoring the wall clock, it passes (that also breaks sleep
detection for the sleepers, so it only says the history changes). So: the forward step of 447 s on m9 at 16.7 s is read by m9's sleep
check as a sleep (its wall gap exceeds the sleep limit): m9 resets its signs of life, skips a commit, and takes a removal on trust for a minute; m9 leaves 20 s later, and the
different history (another committer order, another swap at 59.6 s) ends in the fork. The sleep check is deliberate and documented
(a forward step at most marks the tab as slept); the fork it leads to is H-L2-06. I did not trace the fork's genesis step by step.
Belongs in the failure list under H-L2-06; the comment "which the sleep check reads as a sleep" is true but suggests a different cause than the one that blocks the heal.
3. drift/hostile 20 and extreme/honest 82: harness matters, not defects
- Seed 20, plan
drift(0.5 percent rate error), hostile: "took the removal of m4, who is in the room and was heard 27901 ms ago". The limit is 30 s and the receiver's rule refuses only while the silence is at mostlimit - slack, with a slack of one maintenance interval (2 s in the simulator): it accepts from 28 s by its own clock. With its clock 0.5 percent fast, 28 s on its clock is 27.86 s of real time; 27.9 s is exactly there. The checker's tolerance is 2 s and does not add the drift. So the removal is inside the protocol's rule plus the clock error: the oracle's tolerance should beslack + drift * limit, or the plan's removal tolerance widened; or (honestly) the sentence in SECURITY.md should be "the limit minus one interval" (Part A note 4). It passed before only because the history differed. - Seed 82, plan
extreme(3 percent), honest: "took a commit from m5 although m3 commits now". The plan's header says it is far beyond real hardware and "here to show that the oracle can see a clock fault at all"; with 3 percent a member's patience for a silent lowest id (limit + 2 intervals) ends a second early in real time (34 s against 33 s). It will fail on some seeds whenever the history changes. Not a defect. Either add the seed to the known list with a comment "extreme plan, 3 percent, harness" or give the plan's checker the same drift allowance as above. I did not trace the seed further.
Neither belongs under a finding id; both are harness matters (the oracle's tolerance is not clock-aware) with the SECURITY.md wording note above.
4. liar/hostile 102: a slow tail that H-L2-04's timing moved over the line; cause not traced; no new finding yet
On the merged tree it agrees after 284 s (the limit is 240 s): at 315 s m4#3 still holds an old epoch (30) while the room is at epoch 86 (its
violation text calls it "no group"; it holds keys of its old group). It heals at 359 s. Variants (all with the roster-lying plan on the hostile relay):
merged tree 284 s (fails); without H-L2-04's retry, with the clock fix, 164 s (passes); with H-L2-04 and the clock fix
reverted, fails; neither fix, 164 s (passes). So it is H-L2-04, not the clock fix, that moves it: with the retry switched off, the seed passes.
The new path did not start anybody over in this run (12 holds, each resolved by the package arriving; no start-over at all), so what moved is the timing of
12 commits delayed by 4 s, and the run ends in a different slow tail: a member behind the room for four minutes, receiving commits from the
lowest id (m1#7, a reloaded tab) that it does not take, and not starting over. I suspect (not shown) that "a commit ahead" is only accepted when the
sender's leaf index matches the leaf the member's old tree holds for that id, which fails after the lowest id reloaded, leaving the
member to wait for the heartbeat detector or the zombie leaf's removal. This should become a finding after a reproducer (the question to
answer is why m4 does not start over on the commits ahead from m1#7 between 75 s and 359 s).
Meanwhile it belongs on the failure list (liar/hostile: [102]) as "slow tail (284 s), moved by H-L2-04's timing; cause open", which is
honest, since it passes without H-L2-04 and heals on its own. It is also a harness matter in part: the 240 s of the invariant is
a constant of the compressed timing.
Findings written
findings/H-L2-06.md(in the repository: doc/security/findings/2026-11-hunt/h-l2-06.md): a fork with the same members and the same epoch number cannot be seen by heartbeats; heals only by chance (seeds 5 hostile and 120 honest of the clock sweep).
What I ran
- Part A: the new and reverted tests (all four files, then each file alone),
src/lib/media, ESLint, bothvue-tscprojects. - Part B: the availability sweep, all 8 shards, both relays, merged tree and merged tree without the H-L2-04 retry; a census of holds and start-overs of the new path over 800 runs; traces of hostile 352.
- Part C: seeds 5, 120, 20, 82, 102 with their plans (merged tree); seed 5 and 120 with the wall steps made no-ops, with the sleep mark removed and with the sleep check ignoring the wall clock; seed 5, 120 and 102 for 2 hours; seed 102 with and without each fix; seeds 5, 120, 102 at production ratios.
What I did not check
- A real browser or a real server;
performance.nowbehaviour is from the specification and the code. - lane2-b as a separate checkout (its behaviour was reproduced inside the merged tree).
- The genesis of the forks of seeds 5 and 120 step by step; the cause of seed 102; the production heal time of the equal-epoch fork.
- Whether the oracle's tolerance for removals and patience should be widened in the code (harness work, not done).