Fix review: H-L10-06 (an account erasure aborted as a deadlock)
From mnema-talk v3.3.13, rendered on this site.
What I reviewed: the finding H-L10-06.md, and the branch origin/msch128/fix-h-l10-06 (base origin/main e6c9d9394), which has two commits:
- 87f9b940f: the test, written as a known failure;
- f34d6b5d7: the fix.
I wrote neither the finding nor the fix. The review was read-only: I added nothing to the repository and pushed nothing. Two temporary worktrees, one on the fix and one on 87f9b940f, are removed. The probes were scratch files and were never committed.
Verdict: MERGE. No blocker and nothing to change on the branch. When this lands together with audit-followup, the document items listed at the end need to change with it.
What the fix does
eraseInTransaction (internal/auth/account/account_delete.go:95-124) runs the erasure transaction again when it fails with SQLSTATE 40P01 (db.IsDeadlock, internal/db/errors.go:16-22):
- at most
eraseAttempts= 3 attempts; - a pause of 20 to 100 ms between them (
20 + rand.IntN(80)), ended early if the request ends; - after the third deadlock,
auth.ErrServerBusy: 503 UNAVAILABLE, "the server is busy, try again shortly".
ConfirmIdentity stays before the loop. The 503 is added to the swagger annotation and to api/openapi.json.
(1) Is a rerun safe? I read the whole path myself
Everything up to the commit happens inside the one pgx.BeginFunc transaction:
- eraseWith (account_erase.go:83-102):
SELECT ... FOR UPDATEon users,pinnedOf,runEraseover the merged steps, then the anonymise UPDATE. All of it runs ontx. - Session settings:
mnema.audit_scrubis set withset_config(..., true), which is local to the transaction;- none of the steps uses an advisory lock, NOTIFY or dblink (grep over every file that holds steps and over migrations 0103 and 0104);
- the statement timeout (
db.WithStatementTimeout) is a context value that the pool applies to the connection on acquire, so it is the same for every attempt.
- The audit entry:
Audit.RecordTx(internal/audit/audit.go:78-100) is one INSERT ontx. It reads the actor and request id fromctxand has no side effect. - Triggers inside the transaction, which roll back with it: the guards of migration 0103 (
erased_person_guard,erased_message_stays_empty,users_deleted_is_final,audit_log_erased_person,message_report_erased_people), the message counters, and the migration 0104 column (not touched by the erasure). - After the commit only (deleteAccount, lines 67-79, reached only when
err == nil):removeObjects(object store),announceUnpins(pins_changed events),endDeletedAccount(voice kick, disconnect,OnAccountEnded, the user_disabled broadcast) andSessions.End(cookie). No mail is sent on erasure.
A deadlocked attempt therefore leaves no trace outside the database. Each attempt collects keys and unpins again from the current rows, into a new gone variable inside the loop. One sequence value may be consumed by a rolled-back attempt (the audit_log id); that is harmless.
(2) Does each retry start from a clean state?
pgx.BeginFuncrolls the transaction back when the function returns an error. The next loop iteration callsBeginFuncagain, which acquires a connection and begins a new transaction. No statement and no transaction object is reused.- The deadlock error surfaces through
fmt.Errorf("erase %q: %w", ...)or directly fromCollectRows, anderrors.Asfinds the PgError through the wrapping. - An attempt that finds the account already erased by a parallel request gets
pgx.ErrNoRowsfrom the first SELECT. That is not a deadlock, so it is returned, and deleteAccount answers 204 as before.
(3) The password and 2FA check
ConfirmIdentity runs once before the loop and is never repeated, so a code cannot be replayed through the loop and a lockout is counted once. The loop cannot be entered without a successful confirmation.
NOTE (existed before, unchanged): the factor check spends the TOTP step (UPDATE user_totp SET last_step = $2 WHERE ... last_step < $2, internal/auth/two_factor_store.go:206) or the recovery code in its own commit, before the erasure. After a 503, the web form keeps what was typed. A member who submits again within the same 30-second step with the same code gets a refusal that counts as a failed code. The same was true after the old 500. The message "try again shortly" invites an immediate retry, though. Asking for a new code in the 503 case, or not spending the step until the erasure commits, would be a follow-up; it is not needed for this fix.
(4) The bound, cancellation, the 503 and the client
Bound: three attempts. Two pauses of 20-100 ms. Each attempt is bounded by the bulk statement timeout per statement.
Cancelled mid-loop:
- A cancel during the pause returns
ctx.Err(). - A cancel during an attempt makes its statements fail with a context error, which is not 40P01, so the loop returns it.
In both cases err != nil, so none of the after-commit steps run and nothing was committed. I checked this by reading only; there is no test for it.
503:
- The sentence is in the error catalogue golden file (internal/repo/testdata/error_messages.golden.json), and the internal/repo test passes.
make openapi-check: exit 0.make openapi-diff: "no breaking changes since v3.0.10", exit 0. The 503 is an added response; doc/api-stability.md covers status codes through the OpenAPI document.
Web client:
- The delete form (web/src/settings/components/AccountData.vue:59-86) catches the error and shows
caughtErrorMessage, which maps the sentence toerrors.serverBusy: en "The server is busy. Try again in a moment." (and its German translation in the locale file) resetLocalSessionruns only after a successful erasure.- The bound test confirms that
/api/auth/mestill answers 200 afterwards. - A 503 UNAVAILABLE also fires the client's
onUnavailablelistener. That listener refreshes the server feature list, at most every few seconds (web/src/stores/serverFeatures.ts:68-76). This is harmless.
(5) Reproduction on the base, and my constructed deadlock
On 87f9b940f, where the test still expects the old behaviour, all three cycles reproduce through the real routes:
| Write that closes the cycle | Erasure | Write |
|---|---|---|
| The member's edit, held 2.5 s | 500 INTERNAL_ERROR | — |
| The member's leave of a group they started | 500 | — |
| A pin of the member's message in a Talk they lead | 500 | — |
| The member's edit, held 300 ms | 204 | 500 (the edit is the victim) |
The finding's own probe (review-bundle-integration-deadlock-probe.go.txt), run on the same base, fails at 2.5 s with "the erasure failed with 500".
On f34d6b5d7:
- edit held 2.5 s: the erasure answers 204 and the edit 200. The edit won the first attempt, which was rolled back, and the second attempt erased the edited message.
- leave: 204 and 204.
- Talk pin: 204 and 200.
- edit held 300 ms: 204, and the edit answers 500.
- bound test: three attempts, then 503 with the right code and sentence. The log line names attempts=3.
- the finding's probe passes.
My own canary after the edit won (both words, the original and the edited text, searched in every row of every table): nothing left after the retried erasure. That includes message_edits, which the edit wrote between the two attempts.
My constructed deadlock from the H-L10-05 review (the erased member had timed out the reporter): unchanged. Both orders still abort the report (40P01, the reporter gets 500). The erasure answers 204, the account is erased, and nothing of her text is left. That cycle is closed by the report's insert, so the erasure is not the victim and the new loop is not involved.
In the opposite timing, with the report waiting more than a second before the erasure reaches timeout_by, the erasure would become the victim. That is the same mechanism as this finding, and the new loop now retries it. So the statement "the erasure always wins" in my H-L10-05 round 2 needs that qualification; see the document list below.
(6) Revert check, suites, race, gates
Revert check. With account_delete.go restored to 87f9b940f, four tests fail: edit/2.5s, the bound test, leave, and Talk pin. edit/300ms passes, as expected.
The new tests under -race, all five: pass, and no DATA RACE reported.
Integration suites, all ok, exit 0:
- internal/server: full suite, 325 s;
- internal/auth, internal/auth/account, internal/auth/authn, internal/auth/mailtext, internal/auth/memberadmin;
- internal/db.
Gates, each returned exit 0:
- gofmt;
- go vet, with and without
-tags=integration(auth/..., db, server); make complexity-check;- internal/repo tests;
node scripts/check-english-only.mjs;node scripts/check-doc-links.mjs;- staticcheck v0.8.2, with and without
-tags=integration; make openapi-check;make openapi-diff.
No allowlist entry, no raised limit, no skipped test. Commit types are test and fix. The new function is 30 lines with simple branching, and the complexity gate passed.
Findings
- NOTE: the TOTP step or recovery code is spent before a 503 (section 3). This existed before, unchanged.
- NOTE: no test covers a request cancelled mid-loop. By reading, the after-commit steps cannot run without a commit.
- NOTE: the commit message's alternative (b), locking the member's rows right after the users row, is correctly rejected for a patch release.
Documents when this lands together with origin/msch128/audit-followup
- SECURITY.md "Known open defects": remove the H-L10-06 bullet (audit-followup SECURITY.md:237-243).
- doc/security/findings/2026-11-hunt/h-l10-06.md and its README row (README.md:34): either remove them, or better, keep the file and mark it fixed. If the file is removed, the link to it in doc/security/reviews/2026-11-hunt/bundle-integration.md:299 breaks.
scripts/check-doc-links.mjschecks only README.md, SECURITY.md, doc/.md and doc/security/.md, not doc/security/reviews/** or doc/security/findings/**, so nothing would catch it. That sentence also says "Suggested fix, in a later hunt branch", which needs "fixed by ..." or similar. - The published H-L10-05 review (doc/security/reviews/2026-11-hunt/h-l10-05.md:234, round 2) says "the erasure always wins" about the report cycle. Add: "...when the report closes the cycle; when the erasure is the victim (H-L10-06), it is now retried".
- SECURITY.md (account deletion) and doc/operations.md need no change: "Deletion runs in one transaction" stays true, because each attempt is one transaction. One sentence in doc/operations.md that a deletion meeting the member's own concurrent writes may answer 503 "try again" would be a nicety, not a requirement.
I found nothing else in the documents that needs to change with this fix. audit-followup and fix-h-l10-06 merge without conflicts (git merge-tree).
What I did not check
- A cancelled request mid-loop (by reading only).
- Behaviour under a real two-device timing without the tracer hold.
- The desktop client.