Fix review: H-L5-04 and H-L5-01

From mnema-talk v3.3.13, rendered on this site.

Reviewer: refute-l5. I wrote neither the findings nor the fixes; I refuted and reproduced H-L5-01..05 in phase 2. Both branches are based on msch128/hunt-lane5 at 82e51418f.

BranchCommitVerdict
msch128/fix-h-l5-046e13835dbMERGE AFTER FIXES
msch128/fix-h-l5-012e22972c2MERGE AFTER FIXES

The coordinator asked for the verdicts before every run was done. What is still outstanding is listed under "Not run" at the end.

A. msch128/fix-h-l5-04 (6e13835db)

The design works like this:

  • A post locks its channel row (FOR NO KEY UPDATE) first in messages.insertMessageTx, then draws its number.
  • It writes that number to channels.last_message_number (migration 0103).
  • GET /api/read-state reads the mark in the same statement as the counts.
  • The client:
    • does not count a message_create at or below the mark;
    • adds back the messages above the mark that it counted while the request was on its way;
    • keeps only the newest answer.
  • Followed threads replay the follow changes that arrived while the list was on its way.

Must fix

FIX-1 (BLOCKER for the merge, not for the design): no test detects a missing lock. The commit's test (TestTheReadStateCountsExactlyTheMessagesUpToItsMark) says it does not, and that is true.

Why it misses it: the new UPDATE channels SET last_message_number comes right after the insert. It already serialises the posts of a channel from the insert to the commit. So without the explicit lock the only gap is between nextval inside the INSERT and that UPDATE: a few microseconds, which random writers never hit.

I wrote a deterministic test that hits it: fixreview-H-L5-04-lock-probe.go.txt next to this report, an integration test for internal/server. It works like this:

  1. A test transaction holds a thread root FOR UPDATE.
  2. Post 1 is a reply under that root that mentions the reader. It draws its number, then waits in its INSERT on the foreign-key share lock of parent_id.
  3. Post 2 is a root message in the same channel that mentions the reader.
  4. The reader takes GET /api/read-state.
  5. After the held lock is released, the test checks that the answer's mention_count equals the number of mentioning messages numbered up to its mark.

Results:

  • Without the lock line: fails 3 of 3 runs: answer counts 1 mentions but 2 mentioning messages are numbered up to its mark 3 (one was still open).
  • With the fix: passes 4 of 4 runs.

It needs only the existing harness (newApp, a.db, register, readState). The probe has debug logging of pg_locks, which should be dropped when it goes into the branch. Add it, or an equivalent, to the branch.

FIX-2 (coordinator): migration number. H-L10-05 also adds a 0103.

  • Whichever is renumbered needs its file name and its doc/upgrade.md table row changed together.
  • Nothing else pins the name.
  • The checksum is recorded per file name, so renumbering before release is safe.

Checked, no defect

(1) Every writer takes the lock and writes the mark. INSERT INTO messages appears once in production code (internal/chat/messages/messages.go:365, CreateMessage). CreateMessage has one production caller, insertMessageTx, which insertMessage and InsertMessage wrap. Every path goes through it:

  • Service.Post: the HTTP route, slash commands, bot API, scheduled messages (internal/later/scheduled_send.go:138)
  • webhooks (internal/webhook/handler_store.go:43)
  • uploads (internal/media/handler_upload.go:126)
  • forwards (internal/chat/messages/forward.go:182)
  • forum posts (internal/chat/forum/forum_posts.go:76)
  • polls (internal/poll/service.go:62)
  • Talk invites (internal/chat/talks/talks_invite.go:62)
  • conversation call cards (internal/chat/dm/dm_calls.go:353)
  • bot interaction responses (internal/bot/interactions_respond.go:183)

internal/chat/chattest/member.go:44 is test-only. No code moves a message to another channel or renumbers it.

(1b) The proof. Numbers come from one global identity sequence (with no cache), plus a downward sequence for conversations.

  • Within a channel, the lock is held from before nextval until the commit. So the messages of a channel commit in number order.
  • ReadStateSQL is one statement, so one snapshot. The mark M it reads was written by the transaction that drew M.
  • Every lower-numbered message of that channel drew its number while holding the lock earlier, so it had committed or rolled back before M's transaction got the lock.
  • A higher number would have overwritten the mark had it committed.

The argument holds. Conversations are not in GET /api/read-state (WHERE c.type <> 'dm'), so their downward numbers never become a mark. The Talk chat of a conversation draws public numbers and behaves like a channel.

(2) Migration 0103. It is a new file only; no applied migration is edited.

  • ADD COLUMN ... NOT NULL DEFAULT 0 takes no rewrite (fast default).
  • The backfill is one grouped pass over messages.
  • Migrations run in one transaction each, with no statement timeout, under the advisory lock (internal/db/db.go:250).

Measured with the real migrations 0001..0102 applied in a scratch schema, then 0103 timed (probe in fixreview-H-L5-04-measure-probes.go.txt):

messageschannels0103 durationwrong marks after backfill
1,000,000200109 ms0
10,000,0002004.4 s0
  • Locks. ALTER TABLE holds an ACCESS EXCLUSIVE lock on channels for that time. The documented upgrade stops the app first (doc/upgrade.md: "Stop the app before changing its image or database"). Migrations are forward-only, and newer binaries refuse to start on a database migrated by a newer version.
  • Rolling update. None is supported. If an old binary still wrote messages after the migration (against the docs), its posts would not move the mark. The mark would stay low until the next post from the new binary, and only answers that cross a live event could count a message twice. It heals by itself.
  • Down path. The repo has none; rollback is restoring the backup.

(3) Lock order and deadlocks. The new lock (FOR NO KEY UPDATE) conflicts with other posters, FOR NO KEY UPDATE, FOR SHARE and FOR UPDATE on the row, and plain UPDATE channels. Before the fix a post held only the foreign-key FOR KEY SHARE.

I read every explicit lock and every UPDATE/DELETE on channels and messages:

PathOutcome
Pins (pins.go:174), webhook creation (webhook/store.go:212), outgoing webhooks (outgoing_store.go:181)Lock the channel row first: same order.
Conversation gate (dm_gate.go:92)Upgrades to FOR UPDATE in the same transaction: fine.
L7-06 member add (dm_members.go, branch msch128/fix-h-l7-06)Locks the conversation row first, then reads: same order.
Message delete and edit (message_mutations.go), bulk delete (moderation.go:219), auto-delete (auto_delete.go:221)Lock messages and never the channel row afterwards: no cycle.

One new cycle, found by reading, not run (NOTE): account erasure (internal/auth/account/account_erase.go:82) does three things in order:

  1. locks the user FOR UPDATE;
  2. later runs UPDATE channels SET leader_id = NULL WHERE leader_id = $1 (and dm_created_by);
  3. meanwhile, a post in a temporary Talk led by that user, mentioning them, holds the channel row and waits for FOR KEY SHARE on the user's row (the message_mentions foreign key).

PostgreSQL detects the deadlock after deadlock_timeout and aborts one side: the post or the erasure fails once.

  • It is rare (the Talk's leader erases their account while being mentioned in it).
  • The dm_created_by case existed before, through the conversation gate's lock.

Acceptable for a patch; worth a line in the code comment.

Network calls. None run inside the posting transaction. Automod runs before it. The upload's object is stored before it; inside are only the media row and usage. Webhook files are inserted as rows only.

Throughput. Measured with messages.InsertMessage directly (no HTTP, no rate limit) on a loaded machine (load 8 to 11), two runs each:

writersbefore (no lock, no mark) posts/sp50 / p99after (fix) posts/sp50 / p99
1, one channel364 / 6412.1 / 10.7 ms; 1.3 / 4.5 ms533 / 2481.6 / 5.7 ms; 3.2 / 18.5 ms
8, one channel659 / 6288.3 / 51.9 ms; 8.8 / 54.8 ms480 / 22615.5 / 38.7 ms; 27.5 / 388.6 ms
32, one channel364 / 31964 / 379 ms; 68 / 623 ms348 / 28682 / 256 ms; 93 / 389 ms
8, one channel each569 / 51910.4 / 57 ms; 11.7 / 65 ms345 / 39314.4 / 119 ms; 14.1 / 106 ms
32, one channel each320 / 41565 / 576 ms; 58 / 323 ms319 / 39871 / 464 ms; 63 / 317 ms
  • Busy channel: with 8 concurrent writers in one channel, throughput falls by roughly 25 to 65 % and p50 latency roughly doubles.
  • Spread over channels: within noise. Something global (the per-statement message-count trigger or the pool) limits both sides alike.
  • In practice: a channel still takes a few hundred posts a second, far above what the per-member send rate limit lets real members produce.

(4) No leak. GET /api/read-state drops channels the caller cannot view (handler_readstate.go, visibleReadStates). ViewChannel includes the history, and the mark is the number of a message the caller can already read (every message_create and history row carries it). Conversations are excluded, so the downward conversation numbering of migration 0050 stays private.

Side note: the backfill sets a conversation's mark to max(number), its oldest message, because conversation numbers count down. The post path writes its newest. Both are unused because conversations are excluded. NOTE: say so in the migration comment.

(5) Compatibility.

  • An old client against the new server: the 3.0.9 decoder isChatReadState ignores unknown fields.
  • A new client against an old server: the field is optional; the answer is applied as before. My own case below shows a later event still counts.
  • Two tabs: each tab has its own state. A read in the other tab arrives as read_state with counts, and noteReadStateCleared keeps it against an older answer.
  • OpenAPI: the field is added as optional. The commit says openapi-diff shows no breaking change; I did not rerun contracts-check (see "Not run").

Own cases (temporary vitest on the storeLists harness, 4 of 4 pass):

  1. A mention and a plain message cross a held answer: both are added back, [3, 1].
  2. A thread reply that mentions me, delivered before an answer that includes it, counts once.
  3. A read in another session while the answer is held keeps the read.
  4. An answer without a mark is applied as before, and a later event counts.

Ran on 04

  • Full vitest (3 workers): 669 files passed, 12,065 tests passed, 8 expected fail, exit 0.
  • Revert check: with the client files (events.ts, readState.ts, threadFollow.ts) put back to the base, 5 tests fail: the three crossing cases, the follow case and the trace smoke. Restored afterwards.
  • Trace smoke: passes.
  • Deep replay: 3 traces newly recorded from this server (seeds 700 to 702), 1,013 runs. No H-L5-04 key appears (no drop/early/late unread, mentions or followed). What remains belongs to 01, 02, 03 and 05, which are not on this branch.
  • Integration suites (internal/server, internal/db, internal/chat/...): all ok, exit 0 (internal/server 234 s).
  • My lock test: described above.
  • gofmt clean; English check exit 0.

On a patch release

I do not object to shipping 04 in a patch release:

  • It is additive both ways.
  • The migration takes seconds at 10 million rows.
  • The proof is sound and the lock order is consistent.

The coordinator should still weigh that this is the only fix in the batch that changes the write path of every message, for a finding rated S4 in phase 2. Its cost is the busy-channel throughput above.

B. msch128/fix-h-l5-01 (2e22972c2)

The design: after a delete, a bulk delete, the auto-delete sweep or an edit, readstate.Recount.Tell sends read_state with counts (wire.ReadStateCounts). It goes to online members who may view the channel and whose read marker lies before the changed messages; for an edit, only those mentioned before or after.

Must fix

FIX-1 (BLOCKER): Recount costs one read-state query per online member, and the request waits for all of them.

  • Tell runs on the request goroutine before DELETE answers 204 (handler_messages.go), and the same goes for the bulk delete.
  • It runs ChannelReadState (the full ReadStateSQL for one channel) once per online member who is behind.
  • Measured with readstate.Recount against the real database, every member online and behind (probe in fixreview-H-L5-01-probes.go.txt):
online membersunread backlog in the channelTell durationevents
1,000500.25 s1,000
1,0005,0004.4 s1,000
10,0005006.8 s10,000
10,0005,00039.6 s10,000

The docs size the server for 10,000 members (doc/architecture.md "Unread tracking with 10,000 members"). At that size, deleting one message makes the deleter wait up to 40 s, and one database connection stays busy all that time. Required:

  • Run Tell off the request path, in a bounded worker like the follow sync, after the broadcast so that the order to each connection is kept. Use a context that survives the request: today a client that closes the request cancels r.Context(), and the recount silently stops part-way.
  • Compute all members' counts in one query, or a few batches, instead of one per member.
  • Auto-delete calls Tell once per batch of a run (sweepBatch). Call it once per channel and run.

FIX-2: the sweep path is untested. The tests build AutoDeleter without Recount, so the sweep path never ran under test.

  • My test (TestReviewSweepTellsTheReader in the probe file) ages a mentioning message by 10 days, sets 7-day auto-delete, sweeps with Recount wired to the hub, and expects read_state counts [0, 0].
  • It passes on the branch and fails (no read_state with counts) with the Tell call in auto_delete.go removed.
  • Note that it must move the reader's sign-up back, or the aged message is never unread.

Add it, or an equivalent.

FIX-3 (small): the comment of TestARecountSkipsWhoReadTheChannel says "and neither is one who cannot see the channel", but the test does not check that. Either test it or drop the claim. By reading, the code fails closed: it checks snap.Channel(...).Has(ViewChannel) for each member, and if the snapshot errors nobody is told.

Checked

  • Who is told:
    • online, deduplicated; marker before since; ViewChannel from the snapshot, failing closed;
    • counts from the same SQL and scan as GET /api/read-state, so mutes, blocked authors and suppressed @all/@here count the same;
    • for an edit, mentioned before (read in the edit's transaction) or after (msg.Mentions holds every message_mentions row, so @all and role mentions are covered).
    • Auto-delete with since = cutoff is a superset of who needs it; harmless.
  • Event and compatibility:
    • ReadStateCounts is registered.
    • api/events.json and eventShapes.ts are regenerated per the commit (contracts-check not rerun by me).
    • The web decoder's isStandardReadState accepts the shape.
    • A 3.0.9 client merges it like any read_state with counts ({...prev, ...p}). An old client is safe.
  • Revert check: with the production files put back to the base, TestADeleteOrAnEditTellsTheMembersWhoseBadgeItMovesTheirCounts and TestABulkDeleteTellsTheReadersTheirCountsAgain fail (no read_state with counts). TestARecountSkipsWhoReadTheChannel passes either way, as expected. Restored afterwards.
  • gofmt clean; English check exit 0.

Notes

  • When both land (the known limit): I merged 04 and 01 for a trial.
    • The Go code merges cleanly: scanReadStates carries last_message_number.
    • doc/architecture.md and the two checked-in traces conflict. The traces must be recorded again.
    • The recount reintroduces the 04 crossing: its counts are read after the commit and carry no mark. On top of that, 04's client treats every read_state with counts as "read elsewhere", so an answer in flight is ignored for that channel.
    • The follow-up: add last_message_number to ReadStateCounts and set the client's mark from it. That needs isStandardReadState widened, because it uses onlyFields. A 3.0.9 tab that is still open would reject the new shape with a logged ContractError and drop it: no worse than today.
  • The open channel: a member reading the channel whose marker lags (the read is throttled to every 2 s) can be sent a non-zero count for the channel they are viewing. The sidebar hides badges while reading, so it is cosmetic.
  • Conversations: GET /api/dms is not recounted. This is declared in the commit.

Not run (outstanding when the verdict was asked for)

  • On 01: full vitest, the trace smoke and a deep replay, and the full internal/server integration suite.
  • On both: go vet with and without -tags=integration, make complexity-check, the internal/repo tests, ESLint and vue-tsc on touched files, make contracts-check. File sizes are within limits by count (largest touched: message_mutations.go 363, events.ts 335 lines).
  • The erasure deadlock in A (3) was found by reading, not reproduced.
  • The desktop client repository was not checked against either change.

Round 2: the combined branch msch128/fix-h-l5-01-04 (3fc821281, then 1f207e6cb)

Verdict: MERGE. Every must-fix item of round 1 is done and checked. The findings below are NOTEs.

The branch starts from fix-h-l5-04 and merges fix-h-l5-01, fix-h-l7-04-05, fix-h-l5-02-03, fix-h-l5-05 and fix-h-l10-05. On top of the merges:

  • 7de1d4de7: migration renamed to 0104
  • 845defa5e: my lock test
  • f88d717d4: the cost documented in doc/architecture.md
  • 14a021cf4: the recount rework, with the mark and tests
  • 95e3ae508: traces recorded again
  • 3fc821281: a split to stay under the size and complexity limits
  • 1f207e6cb: a web-only split of the saved and later stores and the read_state types, for npm run lint

Round 1 items

ItemStatus
04 FIX-1, lock testAdded as TestTheReadStateMarkHoldsWhileAnEarlierPostIsOpen. With the lock line removed it fails; with the lock it passes.
04 FIX-2, migration numberNow 0104, after H-L10-05's 0103; the doc/upgrade.md row changed with it. v3.0.9 ends at 0102. The runner applies files in filename order, so an upgrade from 3.0.9 and a fresh database apply the same sequence and end in the same schema (by reading; no schema dump compared).
01 FIX-1, costreadstate.Recounter: Tell only queues; one worker; at most 500 members per statement. Measured below.
01 FIX-2, sweep testAdded (TestTheAutoDeleteSweepTellsTheReaderTheirCounts). It fails with the Tell call in the sweep removed.
01 FIX-3, members who cannot see the channelNow tested (...AndWhoCannotSeeIt). It fails with the permission filter removed.
Recount without a markReadStateCounts carries last_message_number, and the client applies it like an answer (applyChannelCounts).

Recount cost

My probe now drives readstate.NewRecounter with every member online and behind. "Worker" is the time from Tell until the worker is idle; "small" is how long a one-member channel queued right after the big one waits.

Online membersUnread backlogTell (caller)WorkerSmall channel told after
1,000501 µs28 ms26 ms
1,0005001.3 µs189 ms194 ms
1,0005,0005.5 µs2.19 s2.15 s
10,0005001.8 µs1.88 s1.85 s
10,0005,0001.6 µs21.6 s18.7 s
500 (one statement)50,0001.3 µs4.9 s4.9 s
  • The delete and the bulk delete no longer wait: Tell takes microseconds. The worker's times match the author's (1.97 s / 20.5 s).
  • Head of line (NOTE): with one FIFO worker, a big recount delays the corrections of every other channel. Meanwhile a member sees the stale badge, which is exactly the 3.0.9 behaviour, until the recount arrives or they read the channel. A late recount is still correct: it is computed when it runs, and it carries the mark of that moment. Requests for a waiting channel merge, so one channel never has more than one job waiting.
  • Abuse (NOTE): a member can keep the worker busy by deleting their own messages in many big channels. The cost is bounded: one database connection at a time, and at most 256 waiting channels. Further channels are skipped and logged, and their badges then behave as in 3.0.9. I accept that.
  • Database (NOTE):
    • Each statement reads under a snapshot with ACCESS SHARE locks only; no row locks.
    • A 500-member statement takes about 1 s at 5,000 unread messages and 4.9 s at 50,000 (the documented capacity case).
    • From about 150,000 unread messages, a statement would hit the 15 s DB_STATEMENT_TIMEOUT. The recount is then logged, and the badges behave as in 3.0.9.
    • The 2-minute cap per recount works the same way: batches finished before it are sent, the rest is not.
  • Shutdown: I cancelled the context during a 10,000 × 5,000 recount.
    • The worker stopped at once: the running statement is cancelled through its context, and Wait returned in under 1 ms.
    • A Tell after the cancel is refused (r.ctx.Err()).
    • Router.Close cancels first and waits at most jobsStopWait (10 s), so a 2-minute recount cannot hold it.

The batched statement against GET /api/read-state

Probe TestReview2BatchedCountsEqualTheReadState, 25 seeds, each with 41 members on one channel in a category:

  • random channel, category and server notification levels (inherit / all / mentions / mute), with mutes expired or still running;
  • random suppress_everyone;
  • 30 random blocks;
  • 300 messages: 30 % thread replies, 20 % replies to an earlier message, and up to 3 mentions each of every kind (0 name, 1 @all/@here, 2 role).

For every member, ChannelReadStates returned exactly the unread count, mention count, notify level, last read and last_message_number of GetReadStates. 0 differences, and no member left out wrongly.

A 3.0.9 tab that stays open

I ran the real v3.0.9 decoder (a v3.0.9 worktree, a temporary vitest):

  • The round-1 shape (channel_id, last_read_at, unread_count, mention_count) decodes.
  • The round-2 shape with last_message_number throws ContractError: Invalid server response (WebSocket event). The cause is onlyFields in isStandardReadState.
  • handleSocketMessage (v3.0.9 web/src/stores/chat/socket.ts:64-71) catches it per frame, logs WS parse error, and drops that event. Other events go on.

So the author is right. My round-1 sentence "an old client is safe" was about the round-1 shape, which 3.0.9 does accept. The open old tab keeps its old badge (the 3.0.9 behaviour) and logs one console error per recount, until it reloads on the version banner. Nothing else breaks.

The recount's mark against a live message_create

CaseResult
Event of a message above the mark arrives before the countsCounted, kept, added on top. Test: "a new message whose event came before the recount still counts".
Event of a message at or below the mark arrives after the countsNot counted. Test: "a new message the recount already holds does not count again...".
Event at or below the mark arrives before the countsCounted, then filtered out by the mark. Covered by fromAnswer.

With the applyChannelCounts routing removed from events.ts, both tests fail; restored afterwards.

The client keeps at most 500 counted messages and drops the oldest first. Only messages committed after a mark's snapshot can come on top, so the old ones are already covered.

The merge with H-L7-04-05

  • Edit and delete now write their audit entry inside the transaction (the audited and auditDelete callbacks). Every publish (the broadcast, then Recount.Tell) comes after the commit.
  • insertMessageTx still takes the channel lock first, and there is still exactly one INSERT INTO messages in production code.
  • The new erased_person_guard trigger (0103) takes FOR KEY SHARE on users, which the foreign key took anyway. It fires only on changes of person columns, not on UPDATE channels SET last_message_number.
  • The erasure lock cycle of round 1 is now named in a comment in post.go.
  • Deep lane 7 crash sweep (RACE_SWEEP=1 TestEveryFailingStatementLeavesAllOrNothing): pass, 507 subtests, 56 s.

The split commit 1f207e6cb (web only)

It is a mechanical move. I compared the removed and added lines:

  • newestOf and load/refresh/reset moved to laterLoading.ts.
  • The saved store's in-flight bookkeeping moved to savedInFlight.ts and its edits to savedEdits.ts, with the same conditions (loads > 0 for remember and for deleted roots).
  • isReadStateUpdate moved to types/readStateEvents.ts unchanged.

The account generation of the later store moved from module scope to the store instance. Production creates one Pinia (web/src/main.ts:32), so there is one store instance and the behaviour is the same. No production path ever had two instances sharing it. Only tests create several Pinias, where a per-instance counter is the more correct one.

Ran on 1f207e6cb:

  • SAVED_PROPERTY_SEEDS=5000 (seeds 1 to 5,000, four times the default range) together with saved, later, laterLoading, savedInFlight and storeLists: 47 tests pass.
  • npm run lint: exit 0.
  • Full vitest (3 workers): 672 files, 12,099 tests passed.
  • vue-tsc, the English check, make contracts-check and the internal/repo tests: exit 0.

Ran on the combined branch (Go: 3fc821281; Go is unchanged in 1f207e6cb)

  • Full integration (internal/server, internal/db, internal/chat/...): all 13 packages ok, exit 0.
  • Gates: gofmt clean; go vet ./... 0; go vet -tags=integration ./... 0; internal/repo 0; unit tests of readstate, messages and channels 0; make complexity-check 0; make contracts-check 0; English 0; ESLint on the touched files 0; vue-tsc 0; trace smoke pass.
  • Revert checks (each failed as it should, restored afterwards):
    • lock removed: the lock test fails;
    • delete Tell removed: the delete test fails;
    • permission filter removed: the cannot-see test fails;
    • sweep Tell removed: the sweep test fails;
    • client applyChannelCounts routing removed: the two crossing tests fail.
  • Full vitest on 3fc821281: 670 files, 12,089 tests passed.

NOTEs (no change needed for the merge)

  • The edit's candidate list is append(mentioned, msg.Mentions...). It is non-nil only because pgx.CollectRows returns an empty, non-nil slice. Were it ever nil, an edit of a message without mentions would recount every online member of the channel. Making "no candidates" explicit would remove that dependence.
  • Head of line, the statement timeout, the 2-minute cap and abuse: as above. Each degrades to the 3.0.9 behaviour, never to a wrong badge.

Not run in round 2

  • No schema dump was compared between a fresh database and an upgrade from 3.0.9; that is by reading only.
  • The throughput table of round 1 was not measured again; the posting path is unchanged.
  • The desktop client repository was not checked.

The probes of round 2 are in fixreview-H-L5-01-04-round2-probes.go.txt next to this report.

Something wrong on this page? Tell us.