Fix review of H-L7-06 and H-L7-07 (lane 7, phase 3)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: fix reviewer (independent; I wrote neither the findings nor the fixes). Read-only: nothing in the repository was changed, nothing pushed.

Branch msch128/fix-h-l7-06 on 170a9d27a (main after 3.0.9):

CommitWhat
e4c3eb795fix: read a group's members after its lock, so parallel adds keep the cap of ten (H-L7-06, S3)
e73649c08fix: check for the Owner's replies after a delete holds the message's lock (H-L7-07, S4)
31309b47amerge of msch128/hunt-lane7 at d23b69096 (the bulk-delete test and the crash-point sweep)

Inputs: findings H-L7-06 and H-L7-07, phase 2 reports refute-H-L7-04-06.md and refute-H-L6-02-L7-07.md. Method: one temporary worktree (removed afterwards), the built web app copied in so that internal/server compiles, Docker for the integration tests, nice -n 10, -p 1, one heavy job at a time. My probes were a temporary _test.go file, never committed.

Verdicts

FindingVerdict
H-L7-06MERGE
H-L7-07MERGE

No blocker and no should-fix. The notes are below.


H-L7-06: group conversations keep the cap of ten

The change

lockGroupDM (internal/chat/dm/dm_members.go:18) now runs two statements. The first is SELECT dm_key IS NULL FROM channels WHERE id = $1 AND type = 'dm' FOR UPDATE. The second, a statement of its own, reads membership and size: SELECT COALESCE(bool_or(user_id = $2), FALSE), COUNT(*) FROM dm_members WHERE channel_id = $1. Under READ COMMITTED the second statement takes a new snapshot after the lock is held, so it sees what an add or a leave that held the lock before it has committed.

What is locked does not change, and neither does the order. The old statement already locked the conversation row for every caller, members or not, because membership was only a column of the select list. Strangers and unknown ids still get the same 404 "direct message not found" (probe below).

Every writer of dm_members

From git grep for INSERT INTO, DELETE FROM and UPDATE dm_members outside tests:

WriterTakes the conversation lock first?Can it break what the second statement relies on?
Add (AddDMMember, dm_members.go:79)yes (lockGroupDM)no
Leave (LeaveDM / removeDMMember, :109; deletes the conversation when it is empty)yes (lockGroupDM)no
Create (createDMOnce, dm.go:337)no (a new row nobody else can see yet)no
Account erasure (account_erase.go:52 deletes the member rows, :53 sets added_by to NULL, :51 deletes sole conversations)noIt only lowers the count, so the cap holds. A member who is being erased can still have one add go through while the erasure is in flight, the same as before the fix (NOTE 2).
Preferences (hidden_at, favourite_pos; dm_prefs.go, dm_gate.go:136)nono: they change neither membership nor count
Kicksnone exist: DELETE /api/dms/{id}/members/{user} accepts only @me or the caller's own id (dm_handler.go:299)n/a
Bansdo not write dm_members (grep)n/a

So every writer that can raise the count or add a member takes the lock first.

Isolation level

No path that runs these statements changes the isolation level. The only IsoLevel in the code is the read-only permission snapshot (perm/snapshot_load.go:45, RepeatableRead). There is no default_transaction_isolation in the code, the compose files or the migrations, so the PostgreSQL default, READ COMMITTED, applies. A database whose default was raised to REPEATABLE READ by hand would turn these waits into serialization errors (a 500) instead of wrong answers. That is not a supported setting.

Lock orders (unchanged by the fix)

  • add: conversation row FOR UPDATE, then the target's users row FOR NO KEY UPDATE (LockUsers), then the INSERT (key share on the conversation, target and adder)
  • leave: conversation row FOR UPDATE, then the member row and read marker, then (if empty) the conversation and its messages
  • create: users rows of everyone in id order, then the new rows
  • erasure: the member's rows in its fixed step order (userdata.Order*), with no conversation lock before OrderDMChannels / OrderDMMembers / OrderDMCreatedBy

The fix adds no lock and changes no order. The erasure order allows a deadlock that already existed (NOTE 1). PostgreSQL detects it and aborts one side.

Tests and probes

  • The recorded tests pass 5 of 5 (smoke).
  • Deep sweep (RACE_SWEEP=1): 100 of 100 rounds with 2 adders and 100 of 100 with 8, 0 violated.
  • TestAMemberWhoLeftCannotAddAnybodyWithAnAddThatWaitedForTheLeave: 20 of 20 runs answer 404 with nobody added.
  • Revert (dm_members.go back to 170a9d27a): both tests fail, with 11 members (2 adders), 12 to 17 members (8 adders) and "dave in the group: true".

My probes, not in the tests (10 rounds each):

ProbeFixBase
One member sends 5 adds at once to a group of 9 (the finding's "not shown"; the start rate answers some with 429)0 of 10 bad (exactly 10)9 of 10 bad (11 members)
6 adds and a leave at once in a group of 9: count = 9 + adds - leave, never above 100 of 10 bad (for example 409 x4, 200 x2, leave 204, 10 members)9 of 10 bad (13 members)
4 members add the same person at once at 8 members0 of 10 bad (one row, all 200)0 of 10 (unchanged)
A stranger adds to an existing group / an unknown idboth 404 "direct message not found"same

Notes

NOTE 1 (pre-existing, not changed by the fix): an erasure of member U and U's own leave of a group that U created, running at the same time, can deadlock. The erasure holds U's member row (OrderDMMembers) and then wants the conversation for UPDATE channels SET dm_created_by = NULL; the leave holds the conversation and wants U's member row. PostgreSQL aborts one of them, so one request fails once. The lock order is the same before and after the fix. I did not run it.

NOTE 2 (pre-existing): erasure does not take the conversation lock. An add by a member who is being erased at that moment can still go through. The result can be a row whose added_by names the erased account, if the add commits after erasure step OrderDMAddedBy. It does not affect the cap.

Cost: one extra round trip per add or leave, a primary-key range read of at most ten rows. Adds and leaves are rare and rate-limited, so no hot path is affected.


H-L7-07: the Owner check after the message's lock

The change

  • Single delete (lockMessageForDelete, message_mutations.go:139): first SELECT user_id, channel_id, parent_id FROM messages WHERE id = $1 FOR UPDATE, then a statement of its own: "the author is the Owner, or the Owner has replied to this message".
  • Bulk delete (queryLatestMessages, moderation.go:213): the locking query stays as it was. After it, dropOwnerThreads (:242) runs one more read on the locked ids and leaves out every message that has a reply from the Owner. The Owner's own bulk delete (byOwner) skips that read.
  • doc/moderation.md:217 now says that a thread whose reply is committed while the delete runs is also left alone, and that the delete then takes fewer messages than asked (the summary audit entry has requested and deleted).

Every writer that can change what the second statement reads

The second statement reads two things: whether the author's users.role is 'admin', and whether any row has parent_id equal to the message and an author with role 'admin'.

  • New replies. Every message insert goes through one statement (git grep 'INSERT INTO messages'): internal/chat/messages/messages.go:365. Its callers include the reply route, webhooks (the webhook's author is not the Owner), bots, scheduled messages at send time, forwards and the slash commands. The only other insert is the test helper chattest/member.go. Each insert with a parent_id takes the foreign key's KEY SHARE lock on the root. parent_id REFERENCES messages(id) ON DELETE CASCADE (migration 0001) is not deferrable, so that lock is taken at insert time, not at commit. KEY SHARE conflicts with FOR UPDATE, which both delete paths take. Also, no code updates parent_id (grep SET parent_id: only wiki pages).
  • Replies already committed. A reply that started before the delete holds the KEY SHARE lock, so the delete waits and its next statement sees the committed reply. That is the finding's case, fixed. A reply that comes after the delete holds the lock waits, and then its foreign key check finds the root gone. Probe: 400 parent message not found, no row.
  • Nested replies. Threads are flat: only a root has a thread (handler_lookup.go:82), so a check of the direct replies covers everything the cascade removes.
  • The Owner's role. users.role = 'admin' is set only by the seed; no path changes it.

The bulk path: locked, then dropped

dropOwnerThreads runs after the locking query has returned, so every row in gone is locked FOR UPDATE by this transaction until commit.

  • A reply committed before the drop: the drop runs on a new snapshot after the locks, so it sees the reply. That covers a reply that committed while the query waited for some other row, not only the row it waited for (the query locks rows one by one, and a row it locked late may have gained a committed reply before the statement reached it).
  • A reply between the drop and the DELETE: none can land on a row that is still in the list. Such a reply needs KEY SHARE on a row that this transaction holds FOR UPDATE, so it waits until commit and then fails its foreign key check, because the row is gone.
  • Dropped rows: they stay locked until commit. A reply to them simply waits that long and then succeeds. That is correct, because they are kept.
  • DELETE and cascade: they use exactly the filtered ids (TakeLatestMessages, moderation.go:280-295), as do attachments, audit entries and events.

The lock is therefore enough: nothing can change the drop's answer for a row that is deleted.

Isolation level and lock orders

READ COMMITTED, the same as above. Both delete paths take the same locks as before, and the fix only adds plain reads.

  • single delete: the message FOR UPDATE, then the cascade to its replies and media
  • bulk: the member's messages FOR UPDATE in created_at DESC, id DESC order, then the cascade
  • reply insert: KEY SHARE on the root (plus users and channels)
  • edit: the message FOR NO KEY UPDATE (does not conflict with KEY SHARE)

No new order appears, so no new deadlock is possible.

Cost on the delete hot path

There is one extra round trip: an indexed EXISTS (idx_messages_parent_id, migration 0001:76) and a primary-key look-up on users. 200 deletes through the route took 12 to 16 ms each on the fix and 55 ms on the base under higher machine load. The machine noise is far larger than the change, so I can claim no measurable cost, but also no exact number. The bulk delete adds one = ANY read over at most 100 ids.

Tests and probes

  • The recorded test passes 5 of 5 and 20 of 20 (RACE_SWEEP=1, -count=20). The single delete answers 403 with the reply kept; the bulk delete answers 200 {"deleted":0} with root and reply kept. These tests hold a transaction open; they are not sweeps.
  • Revert of message_mutations.go alone: the single delete answers 204 and the reply is gone (FAIL).
  • Revert of moderation.go alone: the bulk delete reports 1 deleted, root and reply gone (FAIL). Each half of the fix is pinned by its own assertion.

My probes, not in the tests:

ProbeFixBase
Bulk delete of 3, the Owner's reply to the middle one held open200 {"deleted":2}; the middle root and the reply kept, the other two gone200 {"deleted":3}, the reply gone
The Owner replies through the route while a delete holds the root400 "parent message not found", no rowsame
The Owner's own bulk delete during their reply200 {"deleted":1}, root and reply gone (the Owner may)same

Notes

NOTE 3: the e73649c08 commit message names a new test, TestAModeratorsBulkDeleteThatWaitedForTheOwnersReplyKeepsIt, and a shared helper. The conflict resolution in merge 31309b47a replaced both with the hunt branch's inline bulkDeleteWhileTheOwnerReplies, inside TestAModeratorsDeleteThatWaitedForTheOwnersReplyKeepsIt. git show --remerge-diff shows the conflict in internal/server/owner_reply_race_integration_test.go. The resolved test asserts the fixed behaviour and fails on revert, so the coverage is equal. Only the message is inaccurate, and with squash merges that message may survive into main: worth one corrected sentence in the squash body.

NOTE 4: the bulk check runs only after the single-delete part of the same test. Because it uses t.Errorf, not t.Fatal, both halves report on their own.


The other statements of the same shape (read only)

Here "a lock wait changes the outcome" means: a statement that waited for a row lock uses a value that another transaction committed during that wait.

PlaceCan a lock wait change the outcome?
internal/chat/channels/auto_delete.go:214 (selectExpired)No wait, but a snapshot race: FOR UPDATE OF m SKIP LOCKED never waits, so no row is checked against a snapshot older than a wait. A root with an open reply transaction is skipped. A reply pinned, or a newer reply committed, after the statement's snapshot but before the cascade is not seen by the NOT EXISTS, and goes with the root. That is a plain snapshot race between the select and the DELETE, not a wait. It is the strongest candidate, as the author says. A pin takes only FOR NO KEY UPDATE on the reply and does not block on the root's lock.
internal/automod/raid.go:60Yes: the join count (a subquery over users) comes from the snapshot before the wait for the automod_state lock. A join that commits during the wait is not counted, so a raid can start one join later than its threshold. active, raid_report and the locked row itself are current.
internal/auth/key_rotation.go:87No for the outcome: every decision column comes from the locked user_totp row, which is re-read after the wait. Only u.username, used in the report text, can be stale. A row inserted during the run is missed whether or not there was a wait.
internal/webhook/outgoing_store.go:288No: the locking subquery returns the current row after its wait, and the outer UPDATE of the same row re-reads the newest version. was_enabled, consecutive_failures and enabled are therefore all current.
internal/chat/forum/forum_tags.go:172No: the subquery locks the tag row itself, so o is the current version after the wait, and the "previous" values and the COALESCE use it.
EditMessageTx (message_mutations.go:44)Yes, narrowly: hasMedia (and isPoll) are EXISTS subqueries in the statement that waits for FOR NO KEY UPDATE, for example behind another edit or delete of the message. If an attachment of the message is removed while the edit waits, an edit to empty text passes the "content, media or forward" check and leaves an empty message. An upload's KEY SHARE does not conflict with FOR NO KEY UPDATE, so adding media causes no wait. Polls are created with their message, so isPoll cannot change. I did not check whether one attachment can be removed on its own.

What I ran

All on the branch head 31309b47a unless stated.

  • Gates: gofmt clean; go vet with and without -tags=integration on internal/chat/... and internal/server; make complexity-check 0; make staticcheck 0; go test ./internal/repo/ ok; node scripts/check-english-only.mjs 0; node scripts/check-doc-links.mjs 0. File sizes: moderation.go 329 lines, message_mutations.go 340, dm_members.go 165. Commit types fix: and test:, no !, no attribution lines.
  • The four tests, -count=5: pass.
  • Deep sweeps: RACE_SWEEP=1 on TestParallelAddsStayWithinTheGroupSize (100 rounds for each adder count), and RACE_SWEEP=1 -count=20 on the leave test and the Owner reply test. All pass.
  • Revert checks: dm_members.go, message_mutations.go and moderation.go each reset to 170a9d27a on its own. The matching assertion fails every time.
  • Probes: the tables above, on the fix and with the three production files at 170a9d27a.
  • Full suite: go test -tags=integration -race -count=1 -p 1 on ./internal/chat/dm, ./internal/chat/messages and ./internal/server: all ok (internal/server 422 s), no data race, no "can't assign requested address". The load average was 14.9 when this run started, above the threshold of 12 in the task, and fell during the run. The run is valid, but not as quiet as asked.

What I did not check

  • The deadlock in NOTE 1 and the auto-delete snapshot race: I reasoned about them, I did not run them. Another agent probes auto_delete.
  • The EditMessageTx attachment-removal path: I read the edit statement only, not whether attachments can be removed on their own.
  • The crash-point sweep commit c246de8a1 that came in with the merge: it is a hunt test, not part of these fixes.
  • The web and e2e suites (the fixes do not touch them).

Something wrong on this page? Tell us.