Fix review: H-L10-01 to H-L10-04
From mnema-talk v3.3.13, rendered on this site.
Phase 3 fix review. I wrote neither the findings nor the fixes. Read-only: nothing changed in the repository and nothing pushed. Each branch was checked out in its own temporary worktree, plus one worktree holding all three branches merged. All of them have been removed. The probe tests below were scratch files and were never committed.
| Branch | Commit | Findings | Verdict |
|---|---|---|---|
| msch128/fix-h-l10-01 | 425838109 | H-L10-01 | MERGE |
| msch128/fix-h-l10-02 | 528ebcb81 | H-L10-02 | MERGE |
| msch128/fix-h-l10-03 | a4c15917d | H-L10-03, H-L10-04 | MERGE AFTER FIXES (one train item) |
Each commit sits directly on the lane 10 test commit 47531d5c2, which sits on 3.0.9. Only the fix commits were reviewed.
Cross-branch item (SHOULD-FIX, train blocker): the three branches merge without conflicts (git merge-tree). After the merge, knownGap (internal/server/personal_erasure_gaps_integration_test.go:20) has no caller left. At 47531d5c2 its seven calls were all in that file: two removed by 01, four by 02, one by 03. staticcheck -tags=integration ./internal/server/ (v0.8.2, the Makefile version) on the merged tree reports func knownGap is unused (U1000), so make staticcheck and CI fail. On each branch alone it is clean. Whichever branch lands last, or the train commit, must delete the helper. That fix belongs on 03 if the order is 01, 02, 03.
H-L10-01: retry ids and upload bookings of an erased account (fix-h-l10-01)
Cause removed: yes. chatEraseSteps (internal/chat/account/account_erase.go:58-62) gains DELETE FROM message_nonces WHERE user_id = $1 and DELETE FROM upload_usage WHERE user_id = $1. They run in the erasure transaction with new order numbers (internal/userdata/order.go:75-76), after the login-unlock step. Both tables are covered by an index on user_id: the primary key of message_nonces (user_id, nonce) and upload_usage_user_idx. So each step is an index scan over one member's rows, which is cheap. Every writer of upload_usage is covered by the one statement: attachment upload, staged upload, wiki images, and webhook files (the webhook files writer books a bot account). The order test (TestAccountErasureKeepsTheOriginalStatementOrder) lists both statements.
Lock order and deadlocks. The erasure starts with SELECT ... FROM users WHERE id = $1 ... FOR UPDATE (internal/auth/account/account_erase.go:82). Every insert into either table checks its foreign key, which takes FOR KEY SHARE on the same users row. FOR KEY SHARE conflicts with FOR UPDATE. So a send or upload either holds that lock before the erasure starts, and the erasure waits for it at its first statement and then deletes the committed rows, or it waits for the erasure. The send transaction writes its nonce after the message insert (internal/chat/messages/post.go:121-127), and an upload writes its usage row after the media insert (internal/media/handler_upload.go:131-138, staging.go:108-114). Both inserts take the users lock first, so a send or upload holds no lock on a nonce or usage row of this person before it competes for the users row. The other writers that lock existing rows are the two pruners and the ON DELETE SET NULL of message_nonces.message_id. The pruners lock only the rows they delete and never need the users row. The cascade fires only on deletion of the person's own messages, which the erasure has already locked at its earlier messages step. I found no lock cycle that involves the two new statements. A deadlock that involves earlier steps would exist without this fix, and PostgreSQL resolves one by aborting a transaction (the erasure answers 500 and can be retried).
Remaining leak in flight (NOTE, existed before this fix, much wider than these two tables). A request authenticated before the erasure commits, whose insert waits on the users row or arrives after the commit, still writes its rows after the erasure. Nothing in the send transaction checks deleted_at. The tombstone trigger users_deleted_final (migration 0043) allows the message_count counter update once disabled_at is set. A SQL-level model of a send in flight, run against the test database, showed this:
- the erasure's FOR UPDATE was held;
- a concurrent
INSERT INTO messages ...plusINSERT INTO message_nonceswas blocked; - then the erasure deleted nonces, emptied messages, set
deleted_atanddisabled_at, and committed; - the blocked insert then went through;
- result:
nonces=1 messages with text=1.
So the nonce row lasts until the pruner runs, up to about 25 hours, as before the fix. A whole message with its text, and a reaction, upload or any other plan table written the same way, lasts permanently. The window is one request's latency at the moment the person erases their own account, for example while sending from a second tab. The cause is that the send and upload paths do not re-check the author inside their transaction. That is outside H-L10-01 and this fix. I report it to the coordinator as a candidate for a new finding (likely S4) and do not hold the fix for it.
The guard (requireEraseStatementsForEmptyTables, internal/server/account_delete_integration_test.go:71-90). It now fails for a foreign key to users that the fixture leaves empty, unless an erasure statement names the column or erasureKeeps gives a reason. Probes I added in a scratch test, each run alone:
- New table
probe_notes(author_id uuid REFERENCES users(id)): fails with "probe_notes.author_id: the fixture has no row and no erasure statement names it". Correct. - Second column on a table the plan already deletes from (
ALTER TABLE message_nonces ADD COLUMN reviewer_id uuid REFERENCES users(id)): passes.erases()(internal/server/personal_schema_integration_test.go:119-129) accepts anyDELETE FROM <table>regardless of the WHERE column, butDELETE ... WHERE user_id = $1leaves rows where only reviewer_id names the person. - Column named like a JSON key in an UPDATE of the same table (
ALTER TABLE message_reports ADD COLUMN user_id uuid REFERENCES users(id)): passes. The snapshot scrub statement contains'user_id'inside a JSON path, so the word-boundary match succeeds. - Real schema: with a stricter matcher (the column must appear as
col =orcol INin a DELETE/UPDATE of the table), no current column changes verdict. The only columns not matched are the bot columns thaterasureKeepslists.
So the guard still passes vacuously for a forgotten second column on a table that is already erased, in a fixture that leaves it empty. Today no real column falls in that gap.
Findings for this fix:
- SHOULD-FIX (non-blocking): tighten
erases()so the column must appear in the WHERE clause (for DELETE) or in SET/WHERE (for UPDATE). With that change the real schema still passes, and both coincidence probes fail. Until then, SECURITY.md:418-419 ("fails when a table that refers to a person is forgotten") is true for tables, not for columns. - NOTE: upload_usage is a media table, but it is deleted from the chat erase steps. It works because the order is global. A media-side registrar would be tidier.
Revert check. With account_erase.go restored to 47531d5c2, four tests fail:
TestErasureRemovesRetryIdsAndUploadBookingsOfThePerson: "1 message_nonces rows and 1 upload_usage rows still name the person".TestEveryColumnAboutAPersonIsErasedOrKeptForAReason: "not covered and no reason is listed", for both columns.TestDeletionLeavesNoPersonalDataInAnyTable: the new guard reports both columns.TestAccountErasureKeepsTheOriginalStatementOrder: "got 67 statements, want 69".
All four pass with the fix. The recorded test is a plain assertion now, no longer knownGap.
Gates. All clean:
- gofmt;
- go vet, with and without
-tags=integration, on the touched packages; make complexity-check;- internal/repo tests;
- English-only check;
- doc link check.
No allowlist entry, no raised limit, no skipped test. erasureGaps became empty, which is a list shrinking, not a gate weakened. Commit type fix.
Suites (full, integration tag). internal/server ok (245 s). internal/userdata ok. internal/chat/account has no tests.
Verdict: MERGE. The matcher tightening is a follow-up, and the in-flight insert race is a new candidate finding.
H-L10-02: export of DM settings and comment mutes (fix-h-l10-02)
Cause removed: yes, for the scope the refuter confirmed.
exportProfile(internal/auth/account/account_export_build.go:178-203) selects and writesdm_policyanddm_mute_allinto profile.json. The change is additive and the shape of existing keys is unchanged.- The new file
event_comment_mutes.json(internal/communityevents/accountdata/accountdata.go:101-109) listsevent_id. It is built withrowsFile/FetchRows, like its neighbours, at order 232 (between comments and wiki revisions). - Events and series the member created move to
exportOmitswith a reason, which the refuter classed as a documented decision. The README and doc/events.md:85 now say so.
Probes of my own:
- Row limit and truncation. In the account package, with
exportMaxRows = 3and four muted events (inserted by SQL), event_comment_mutes.json held 3 rows and truncated.json held{"files": ["event_comment_mutes.json"], "limit": 3}. Correct. - Default values. For a member who never changed the settings, profile.json holds
"dm_policy": "everyone"and"dm_mute_all": false. Values are always present, never null. - README lists every file. I built a full export through the API and checked each archive file name against README.txt. All 26 files besides README.txt are named, including event_comment_mutes.json.
- Other members' data in the mutes file. The file holds only event ids from the member's own mute rows (
WHERE user_id = $1). The table has no other columns (migration 0087: event_id, user_id), and its rows go with their event (ON DELETE CASCADE), so a deleted event leaves no id. An event the member can no longer see, for example after losing access to its Talk, still appears by id. That is consistent with event_comments.json and rsvps.json, which do not filter by visibility either. An opaque id the member once had access to is not another member's personal data. NOTE only. - doc/events.md:57 now reads "the export holds the comments in
event_comments.jsonand the mutes inevent_comment_mutes.json". True. doc/operations.md:1648 names the comment mutes.
Findings for this fix:
- NOTE: the package comment (accountdata.go:1-2, "RSVPs, sign-ups and comments") and the
Registercomment (:18) were not updated for the mutes. TheExportcomment was. - NOTE: event_comment_mutes has no timestamp column, so the file has no time. Nothing is left out.
- NOTE: the archive format is not on the public surface of doc/api-stability.md, so no stability entry is needed.
Revert check. With account_export_build.go and accountdata.go restored to 47531d5c2, three tests fail:
TestExportHoldsDirectMessageSettingsAndCommentMutes: all three properties.TestEveryTableAboutAPersonIsExportedOrOmittedForAReason: "event_comment_mutes is not covered", "dm_policy ... not covered", "dm_mute_all ... not covered".TestArchiveFilesStayInTheOriginalOrder: "got 25 files".
The mute check strings.Contains(archive, ev.ID.String()) could in principle be satisfied by another file. The revert shows that no other file holds the event id today, so the test is not vacuous.
Gates. All clean (same list as for 01). exportGaps and profileGaps became empty, exportOmits gained two reasoned entries (lists, not gates). Commit type fix.
Suites. internal/auth/account ok, internal/communityevents/... ok, internal/userdata ok, internal/server ok (250 s).
Verdict: MERGE.
H-L10-03 and H-L10-04: orphan cleanup of exports/, guide prefixes (fix-h-l10-03)
Cause removed: yes.
orphanPrefixes(internal/media/orphans.go:26) gainsexports/.orphanLookup(:73-78) checksexports/keys againstdata_exports.s3_keyand the four media spaces against media rows as before.sc.lookupis set per prefix, and the batch is flushed at the end of every prefix (:64-66). So a partial batch of one prefix is never checked with another prefix's query.
This covers every variant in the finding and the refuter's report:
- a failed DeleteBatch after the erasure;
- a crash between the erasure commit and the delete;
- the build that finishes after the erasure deleted its row (account_export_build.go:131-137, whose error is dropped);
- a failed delete after a failed build.
Can it delete an export that is still wanted? No, as long as the row exists. Write order in buildExport (account_export_build.go:103-138): the object is uploaded first, then UPDATE data_exports SET status='ready', s3_key=.... The build runs under exportBuildTimeout = 5 minutes (account_export.go:25, :128). So an object without its row is at most 5 minutes old, well inside the 24-hour age rule (orphanMinAge).
Object age comes from the store's LastModified in ListObjectsV2 (internal/s3/s3.go:230), compared against the app's clock. The export upload is a multipart upload (internal/s3/stream.go:75). Stores differ on whether LastModified is the initiation or the completion time. I did not check this against SeaweedFS or other stores. With either meaning, it lies at most one build (5 minutes) before the row is written, so the rule holds. An incomplete multipart upload is not listed and so not touched. Clock skew between app and store would have to be close to 24 hours to matter.
Once written, the row keeps the key until the object is gone:
dropExportFiledeletes the object first and clears the key only on success (account_export.go:288-296);- the sweep goes through
dropExportFile; - rows are pruned only when
s3_key IS NULL(:347).
Probe of my own (API level, objects 48 hours old):
- Kept (a row names each one):
- a downloaded export whose delete had failed (row keeps the key);
- an expired export the sweep has not reached;
- a failed row that still holds a key.
- Deleted (
Count:2 Deleted:2):- an object of a pending row without a key;
- a stray
exports/not-an-export.txt.
- Dry run:
GET /api/admin/media/orphansfirst returnedCount:2 Deleted:0and removed nothing. - Access: a member got 403 on both routes. Both are Owner-only (internal/media/handler.go:121,
auth.RequireOwner). The cleanup answers 409 during an online backup (TryOutsideBackup). Nothing deletes on its own: the daily job only counts.
The author's own test covers another member's live export with its row (kept), a young archive without a row (kept), and the left-over upload and export (both deleted). The author also reports that a version checking exports/ against media rows deletes the live export. I did not rerun that variant; the lookup switch is exactly what prevents it.
Restored backup with a database older than the object store. Exports built after the database snapshot have no rows after a restore. Once their objects are 24 hours old, the Owner's cleanup deletes them. That is acceptable for three reasons:
- such an export cannot be downloaded, because
claimDownloadneeds the row (account_export.go:261-270); - after 24 hours it would have expired anyway;
- uploads/ objects in the same situation have no media rows and have always been deleted the same way.
The online backup freezes storage before the database snapshot, so there the only case is an export whose upload finished just before the freeze while its UPDATE came after. That export is pending in the restored database, the sweep marks it failed, and its object becomes an orphan. Correct. The orphan section of doc/operations.md describes the rule generally ("no data export row") but says nothing about restores. That is acceptable: the uploads/ case was not documented either. NOTE.
Behaviour change for shared buckets (NOTE). Objects of other software under exports/ in the same bucket were never touched before. They are now deleted once 24 hours old when the Owner runs the cleanup. The guide states the new rule ("anything outside those five prefixes are never touched"), and exports/ was already the app's own key space, where any foreign object could collide with the app's writes. No upgrade note exists. The commit subject will appear in the changelog. I consider that enough for a patch release.
Documentation.
- doc/operations.md "Orphaned media" now names all five prefixes and what refers to each, which fixes H-L10-04 ("two" after three names, without
covers/). The guide test is a plain assertion now. - The SECURITY.md parenthesis ("the orphan cleanup removes only an archive no export row names") is true; the 24-hour rule applies on top.
- The account-deletion paragraph of doc/operations.md:1658-1660 and the
removeObjectscomment are true now.
SHOULD-FIX: swagger description. The @Description of GET /api/admin/media/orphans (internal/media/handler_admin.go:152) says "Objects in storage (uploads/, avatars/) that no media row refers to". It was already stale (banners/, covers/ missing) and now also omits exports/, whose reference is not a media row. The cleanup description (:177), "Never touches an object a media row refers to", is incomplete for exports/. The published API reference (api/openapi.json) therefore tells an Owner less than the guide does. The fix: update both descriptions and run make openapi. A description change is not breaking for openapi-diff. Not blocking on its own. make openapi-check passes on the branch as it is (exit 0, no diff), because the generated file matches the unchanged annotations.
NOTE: exportKeys (WHERE s3_key = ANY($1)) has no index on data_exports.s3_key. The table holds at most a few rows per member (rows are pruned after 7 days), and the cleanup is a manual Owner action. No cost on a hot path.
Revert check. With orphans.go restored to 47531d5c2:
TestAFailedDeleteOfAnErasedExportIsFoundByTheOrphanCleanupfails ("{Count:1 Deleted:1}, export stored: true");TestEveryKeySpaceIsScannedOrHasAReasonfails ("keySpaces says "orphan scan", orphanPrefixes disagrees");TestTheGuideNamesThePrefixesTheOrphanScanCoversfails ("the guide names [... exports/ ...], the scan covers [...]").
All pass with the fix.
Gates. All clean: gofmt, go vet (both tag sets), complexity check, internal/repo, English-only, doc links, make openapi-check. No allowlist entry. Commit type fix.
Suites. internal/media ok, internal/server ok (283 s).
Verdict: MERGE AFTER FIXES:
- If this branch lands last of the three: delete the unused
knownGaphelper (the train blocker above). - SHOULD-FIX, may also be a follow-up: correct the two swagger descriptions and regenerate api/openapi.json.
What I ran
All runs used nice -n 10, -p 1, one heavy job at a time, and Docker through OrbStack.
Revert checks: 01 (four tests), 02 (three tests), 03 (one server test and two media tests).
Probe tests (scratch files, not committed):
- 01: four probes of the guard and its matcher, plus the SQL model of a send in flight;
- 02: truncation and defaults in internal/auth/account, README coverage in internal/server;
- 03: one cleanup probe in internal/server.
Full integration suites:
- 01: internal/server, internal/userdata;
- 02: internal/auth/account, internal/communityevents/..., internal/userdata, internal/server;
- 03: internal/media, internal/server.
All passed. No "can't assign requested address" failures occurred.
Gates per branch: gofmt, go vet with and without the integration tag,
make complexity-check, internal/repo tests,node scripts/check-english-only.mjs,node scripts/check-doc-links.mjs.make openapi-checkwas run on 03 only, the one branch near swagger.Merge:
git merge-treeof all three branches, then staticcheck v0.8.2 with-tags=integrationon internal/server, on the merged tree and on 03 alone.
What I did not check
- The LastModified meaning for multipart uploads on SeaweedFS, MinIO or AWS. The argument above holds for either meaning.
- The in-flight insert race through the real HTTP send path with real timing. Only the SQL-level model was run.
- A cleanup run against a real restored backup.
- A full staticcheck of the whole module, and
make check. Both are excluded by the hunt rules for this machine. - The web side, which none of the three branches touches.