Phase 2 and fix review of H-L7-09 (and phase 2 of H-L7-10)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: fix reviewer (independent; I wrote neither the findings nor the fix). Read-only: nothing in the repository was changed, nothing pushed.

Inputs:

  • Findings H-L7-09 and H-L7-10.
  • Tests on msch128/hunt-c-l7-b (edaff4797).
  • Fix msch128/fix-h-l7-09 88c1dd590 (one commit on edaff4797).

Method:

  • Temporary worktrees, removed afterwards, with the built web app copied in.
  • A local, unpushed merge of msch128/fix-h-l7-06 into the fix branch (clean, no conflicts), to check the lock orders together with the H-L7-06/07 fixes.
  • Docker for the integration tests; nice -n 10, -p 1, load below 12 before every heavy job.
  • My probes were temporary _test.go files, never committed.

Verdicts

ItemVerdict
H-L7-09, phase 2STANDS (S4). The real window is wider than the finding says: see Reach.
H-L7-10, phase 2N confirmed: no route removes an attachment row from a live message.
Fix 88c1dd590MERGE

Phase 2: H-L7-09

Reproduction

The finding's command on edaff4797, three runs, all PASS with the known-failure line:

go test -tags=integration -count=1 -v -run 'TestAReplyPinnedDuringTheSweepSelectIsDeletedWithItsRoot' ./internal/server/
-> "pin committed during the sweep; pinned reply kept false, root kept false"   (3 of 3)

The oracle is sound. The test holds the pin's UPDATE (the statement of writePin) uncommitted and lets it commit just before the sweep's DELETE. It then counts the pinned reply in the database. The control (a pin committed before the sweep) is kept.

In the same runs, the old TestAReplyPinnedDuringTheSweepIsNotDeletedWithItsRoot passed with "pin during the sweep answered 404; reply kept false". That confirms the finding's point that this test did not exercise the rule.

Documented or intended?

No. doc/moderation.md says "Pinned messages stay, and so does a thread root while it has a reply that is newer than the period or pinned". doc/upgrade.md:645 and the sweepBatch comment say the same. Nothing describes a window in which a pin answers 200 and the message is still removed.

Reach (measured)

The finding puts the window "between the sweep's SELECT and its DELETE", with the pin committing inside it. It is wider than that:

  • The pin holds the reply's row lock from its SELECT … FOR NO KEY UPDATE until it commits. While it does, the sweep's SKIP LOCKED skips the reply and chooses the root.
  • After that, the commit may come at any time. If the pin is still open when the DELETE runs, the cascade waits for the pin and then deletes the pinned reply.

Probe through the real route (a pin with PUT …/pin, its transaction held by a lock on the pinner's user row, the sweep started, the pin committed only after pg_stat_activity showed the sweep's DELETE waiting): on edaff4797 the pin answered 200 and the pinned reply was lost in 5 of 5 rounds.

With no artificial hold at all, I raced a pin through the route against Sweep(), with a small offset that varied by round (0 to 11 ms). On edaff4797, 26 of 300 rounds lost a pinned reply that had answered 200.

MeasurementValue
Selection start to DELETE start, batch of 200 (1000 expired, 5 batches)1.1 to 3.4 ms
The DELETE itself12 to 14 ms
A pin through the route, end to end (20 runs)4.3 to 9.9 ms

So the race needs a pin of an old reply in an auto-delete channel within about 10 to 25 ms of that channel's sweep. The sweep runs every ten minutes. No setup by an attacker is needed, but the coincidence is rare. A moderator cannot widen the window, and nobody gains a right.

S4 stands: a stated rule fails under unusual timing. The finding rated it right; only its window description is too narrow.

Phase 2: H-L7-10 (rated N by the coordinator)

Confirmed: no route reaches it today. hasMedia is EXISTS (SELECT 1 FROM media WHERE message_id = …) and counts rows whatever their is_deleted. These are all the writers that remove or detach media rows (git grep for DELETE FROM media and UPDATE media):

WriterRemoves a row of a live message?
DELETE /api/media/{id} (media/admin.go:99) and the retention prune (:175)No: UPDATE media SET is_deleted = TRUE. The row stays, so hasMedia does not change.
Unstaging and staging cleanup (staging.go:158, :180)Only staged rows, not attached to a message yet.
Event covers (channel_media_keys.go:27)Not message attachments.
Account erasure of sole conversations (account_erase.go:11, :13)The whole conversation goes.
Erasure of the uploader's own media (account_erase.go:30)The author is the account being erased. The same transaction also blanks their messages, so an edit that waited for it ends where the erasure ends.
Attaching (attachments.go:43, handler_upload.go:133, webhookfiles.go:32)They add rows. The upload's KEY SHARE does not conflict with the edit's FOR NO KEY UPDATE, so it causes no wait, and at worst a refused empty edit.
Polls (poll/store.go:30)Created together with their message.

NOTE A (not a race, not a finding): because hasMedia counts soft-deleted rows, an author can edit to empty text a message whose only attachment the Owner deleted. The message is then empty text plus the "deleted attachment" placeholder. That seems consistent with the placeholder design. I only mention it so that a future fix of H-L7-10 decides whether is_deleted should count.


Fix review: 88c1dd590

The two changes

  1. The pin locks the thread root. writePin (pins.go:177-186) takes FOR KEY SHARE on the thread root, after the channel lock and before it locks the reply. For a root, parent_id is NULL, so the statement locks nothing.
  2. The sweep re-checks after its locks. After the locking selection, selectExpired calls dropKeptThreads (auto_delete.go:245). That is a statement of its own, on a new snapshot: SELECT DISTINCT parent_id … WHERE parent_id = ANY(batch) AND (created_at >= cutoff OR is_pinned). The roots it finds are dropped from the batch.

Does it remove the cause, for every variant?

  • Pin in flight when the sweep selects: the pin holds KEY SHARE on the root, so SKIP LOCKED skips the root. The next run sees the pin. This covers the finding's case and my wider one.
  • Pin committed between the selection's snapshot and its lock on the root: the re-check sees it.
  • Pin that starts after the sweep locked the root: its KEY SHARE waits for the sweep. The subquery that finds the root reads parent_id, which never changes. The locked root is then deleted, so no row is locked, the next statement finds no reply, and the pin answers 404. That is correct: the pin never happened.
  • Young reply: in flight, its foreign-key KEY SHARE makes the sweep skip the root. Committed after the snapshot, the re-check sees it. Started after the lock, it waits and fails with "parent message not found" (probe: 400). That is the same as before.
  • Other writers of is_pinned: only writePin sets it (git grep). Account erasure only sets it to FALSE, which can only make the sweep keep less.
  • Replies in the batch itself: a committed pin of a selected reply fails the NOT m.is_pinned re-check under FOR UPDATE, as before.

Nothing can commit between the re-check and the DELETE. The argument in the commit message holds:

  • The batch's roots are FOR UPDATE until commit.
  • A reply insert and a pin both need KEY SHARE on the root first, so both wait.
  • An unpin also takes the root lock, and could only keep more anyway.
  • created_at never changes.
  • The only other change to the decisive rows would be an UPDATE of a reply's is_pinned, and its only writer takes the root lock first.

Lock orders and deadlocks

The pin's new order is: permission advisory lock (asModerator), then the channel FOR NO KEY UPDATE, then the root KEY SHARE, then the reply FOR NO KEY UPDATE, then the user row of pinned_by (KEY SHARE). The new wait edge is "the pin waits for a root". It can close a cycle only with a transaction that holds a root FOR UPDATE and then waits for the channel row in a conflicting mode, or for the permission lock.

Other path (with the fix-h-l7-06 bundle)OrderCycle with the pin?
Single delete (DeleteMessageAs, H-L7-07 fix)message FOR UPDATE, then its replies (cascade)no: it never waits for the channel or the permission lock, and the pin holds no reply before it holds the root
Bulk delete (H-L7-07 fix)permission lock first, then the member's messages FOR UPDATEno: both take the permission lock first, so they run one after the other
SweepSKIP LOCKED on roots and replies, then the cascade on replies; the outbox insert takes KEY SHARE on the channelno: it never waits for a root, and KEY SHARE on the channel is compatible with the pin's FOR NO KEY UPDATE
Unpinthe same path as the pinthe same order, no cycle
Pin of a rootchannel, then the root FOR NO KEY UPDATE (the root-lock statement locks nothing)the same order as before
Channel delete, DM leave, erasure of sole conversationschannel FOR UPDATE first, then the cascadethe channel conflicts first, so the order is the pin's
Account erasureUPDATE of the member's messages (FOR NO KEY UPDATE), …, UPDATE channels SET dm_created_bythe root edge: FOR NO KEY UPDATE is compatible with KEY SHARE, so no new cycle. A cycle over the reply and the channel row predates the fix (NOTE B).

Stress (racetest.Sweep). In each round 8 callers are released together on one thread with jitter:

  • a pin of an old reply;
  • an unpin of a pinned reply;
  • a new reply;
  • a single delete of a third reply;
  • a bulk delete of that third member's messages;
  • a pin of the root;
  • two sweeps.

Moderators rotate over 240 accounts, so the pin limit of 60 per minute is not hit. Deadlocks are counted with pg_stat_database.deadlocks.

RunRoundsViolationsDeadlocks (40P01)Outcomes
Fix + fix-h-l7-06 bundle, 8 callers300000pin 200 x636 / 404 x2364, unpin 200 x3000, reply 201 x2325 / 400 x675, delete 204 x665 / 404 x2335, bulk 200 x3000, root pin 200 x2325 / 404 x675; no sweep errors
Fix + bundle, pin against sweep only3000001500 pins answered 200, all 1500 replies kept
Fix branch alone, 8 callers (48 moderators, so 2040 pin calls got 429)300000no sweep errors
Fix branch alone, pin against sweep3000001474 pins answered 200, all kept
Base edaff4797, pin against sweep300260231 pins answered 200, 26 replies lost

Violations checked after every round:

  • a pin of the reply answered 200 but the reply is gone;
  • a pin of the root answered 200 but the root is gone;
  • a pinned reply remains while its root is gone;
  • a young reply created in the round is gone.

I also checked one pre-existing order that I had suspected: a bulk delete of a member's old root plus reply against the sweep, 600 rounds on the bundle. 0 deadlocks; every bulk delete answered 200.

Starvation

The re-check uses the same conditions and the same cutoff as the selection's NOT EXISTS. Without a concurrent commit it drops nothing. A kept thread is therefore excluded by the selection itself every run: it is never chosen and then dropped, and it cannot use up batches.

Measured: 1000 kept threads (500 with a young reply, 500 with a pinned old reply), older than 1000 expired messages. One run deleted all 1000 expired messages in 5 full batches and kept all 1000 roots (113 ms). The 5 re-checks took 3.5 ms in all. A batch shortened by the re-check ends the run early only when a pin or reply really raced it, and the next run continues.

Cost

Three runs of 5000 expired messages (25 batches) took 524 to 581 ms each. The re-check statements took 11.8 to 13.7 ms in all, about 0.5 ms per batch of 200: roughly 2 % of a sweep. The extra statement on each pin is one indexed point lock on the root.

Tests

  • The recorded tests pass 5 of 5: TestAReplyPinnedDuringTheSweepIsNotDeletedWithItsRoot, TestAReplyPinnedAfterTheSweepSelectIsKeptWithItsRoot, and TestARootWithAReplyInFlightIsSkippedBySweepAndKeptAfterwards.
  • The rewritten route test is stronger, not weaker. The old version passed on the base with "pin answered 404": the sweep simply won, and it asserted only "if 200 then kept". The new version pins through the route and holds the pin uncommitted after its UPDATE (it waits in pg_stat_activity on the pinner's user row). It then requires the pin to answer 200, the reply to be pinned and kept, the root kept, and an unrelated old message deleted (proof that the sweep ran).
  • Revert of change 1 (pins.go at edaff4797): the route test fails ("pinned reply kept false, root kept false"); the other two pass.
  • Revert of change 2 (auto_delete.go at edaff4797): TestAReplyPinnedAfterTheSweepSelectIsKeptWithItsRoot fails ("pinned reply kept false, root kept false"); the route test passes.
  • Each half is therefore pinned by its own test.
  • The second test models a pin that commits between the selection's snapshot and its lock. It does this with the bare UPDATE (without the root lock), committed by a hook just after the selection. That is honest about what it simulates: with change 1, a real pin can only hit that window if it starts and commits completely within those microseconds.

Gates and docs

  • gofmt clean; go vet with and without -tags=integration on internal/chat/... and internal/server; make complexity-check 0; make staticcheck 0; internal/repo ok; English check 0; doc links 0.
  • File sizes: auto_delete.go 279 lines, pins.go 234, the two test files 139 and 231.
  • Commit type fix:, no !, no attribution.
  • The doc/moderation.md addition ("also one that is written or pinned while a run is underway") is true.
  • doc/upgrade.md:645 is unchanged and still true.
  • No REST shape changed. The only behaviour visible to clients: a pin or unpin of a reply whose root the sweep or a delete is removing at that moment now waits and answers 404 instead of 200.

Notes

NOTE B (pre-existing, not changed by the fix): in a group conversation created by member U, U's account erasure and another member's pin of U's reply can wait for each other. The pin holds the channel and wants U's reply; the erasure holds U's reply (UPDATE of U's messages) and later wants the channel (UPDATE channels SET dm_created_by = NULL). PostgreSQL aborts one side. I did not run it, and the root lock does not take part in it.

NOTE C: the root lock is also taken for an unpin, where it is not needed (an unpin can only make the sweep keep less). It costs one point lock and keeps the code simple. Leave it.

What I ran

  • Phase 2:
    • the finding's command 3 times on edaff4797;
    • the commit-during-DELETE probe through the route (5 of 5 lost on the base, 0 of 5 on the fix);
    • pin against sweep, 300 rounds on the base (26 lost);
    • the window and pin-latency measurements;
    • the H-L7-10 writer inventory (read only).
  • Fix:
    • the recorded tests 5 times;
    • revert checks for each half;
    • the stress runs in the table (fix alone and bundle, 3000 rounds each);
    • starvation and cost probes;
    • the bulk-against-sweep check;
    • gates.
  • Full suite: go test -tags=integration -race -count=1 -p 1 on ./internal/chat/channels, ./internal/chat/messages, ./internal/chat/dm and ./internal/server, run on the local bundle (fix + fix-h-l7-06): all ok (internal/server 313 s), no data race, no "can't assign requested address".

What I did not check

  • The deadlock of NOTE B (reasoned, not run).
  • The full suite on the fix branch without the bundle. Only the targeted tests, the stress runs and the gates ran there.
  • The other probes in edaff4797 (raid, delivery, tags), which are not part of this fix.

Something wrong on this page? Tell us.