Fix review: H-L7-04 and H-L7-05

From mnema-talk v3.3.13, rendered on this site.

Reviewer: independent. I ran phase 2 for both findings and wrote neither fix. Read-only. Branch msch128/fix-h-l7-04-05, based on msch128/hunt-lane7 at d23b69096, with these commits:

  • a005c3941, the H-L7-04 fix
  • cb3320716, the H-L7-05 fix (parent d23b69096)
  • b80b21dd6, the merge of the two
  • dc8adbe77, docs

Every diff was read in full (35 files), with the surrounding code. Temporary worktrees were removed. One heavy job at a time, nice -n 10, -p 1, Docker via OrbStack; load stayed between 4 and 12.

Verdicts

  • H-L7-04 (a005c3941 + dc8adbe77): MERGE AFTER FIXES. Two documentation fixes, S1 and S2 below. The code is right.
  • H-L7-05 (cb3320716): MERGE.

What I ran

RunResult
go build ./..., go vet ./..., go vet -tags=integration ./..., go vet in loadtest/viewer, gofmt on every changed Go fileexit 0, clean
RACE_SWEEP=1 go test -tags=integration -run TestEveryFailingStatementLeavesAllOrNothing ./internal/server/ (deep)PASS in 69 s. All H-L7-05 lines are gone. The only known partial writes left are the nine /api/admin operations, as H-L7-04 (assign, create and delete a role, ban, create a category, a channel, a webhook and an invite, delete a channel). 0 "not on the known list".
Full go test -tags=integration ./internal/server/ok, 237 s
go test -tags=integration over internal/chat/..., communityevents/..., wiki/..., webhook/..., later/..., auth/...all ok
Unit tests: internal/repo, internal/chat/messages, internal/auditok
scripts/complexity-check.sh, make docs-check, scripts/check-english-only.mjsexit 0
Revert check H-L7-04: the production code of cb3320716 alone, with the final crash testFAIL as required: 18 partial writes not on the known list (create a wiki page 2, create an event 1, delete a message 1, delete an account 2, edit a message 10, register 1, report 1)
Revert check H-L7-05: the production code of a005c3941 alone, with the final crash testFAIL as required: 36 partial writes not on the known list (answer an event 7, comment on an event 1, create a webhook 1, edit a message 8, open a forum post 8, post a message 8, react 1, start a conversation 2)
Odd-input probe (my own, not committed), see (1)PASS
Latency, base d23b69096 against the fix, two runs each, see (2)no regression

(1) The availability trade: a failed entry now fails the change

  • Intent. SECURITY.md:281 ("every change is audited") and doc/moderation.md ("An append-only record of what was done") ask for the change and its entry together. The new section "When an entry can be missing" states it: "a failed entry fails the change". Failing closed is the documented intent.

  • Can an ordinary action now fail because of the audit insert? I checked every cause I could find:

    • The immutability trigger (migrations 0027, 0043) fires on UPDATE only. The new INSERTs are untouched, also inside the account erasure's mnema.audit_scrub transaction: the auth.account_delete entry is inserted after the scrub, in the same transaction, and carries no name.
    • Size. SanitizeDetails never fails (audit/sanitize.go:44-65): lists are halved, then request/after/before/params are dropped, then the result is {"truncated": true}. Content is cut at MaxContentRunes = 4000 runes, and clip cuts by runes (textcut.Runes), so no invalid UTF-8 can reach PostgreSQL.
    • NUL bytes cannot occur: message text is TEXT, which already rejects them, and JSON keeps \u0000 escaped as text.
    • action is checked to 1-64 characters, and all actions are constants. actor_id is nullable; RecordTx fills it from Log.Actor when it can.
    • A nil *audit.Log returns nil in RecordTx (audit.go:78-81), so the dropped h.Audit == nil checks are safe.
  • Probe with large and odd inputs (through the real routes):

    • a 3999-rune message of ZWJ emoji sequences, edited 60 times, alternating with text that holds U+202E, stacked combining marks, an invalid UTF-8 byte sequence, a literal \u0000 and 100 tag characters;
    • 30 replies, then a moderator delete with a 200-emoji reason;
    • a message of a member who then erased the account: edited before the erasure, deleted by the Owner after it;
    • a report with the odd text as its note.

    Result: every edit 200 and every delete 204. 60 message.edit entries, 1 message.delete. The erased member's message was deleted with 204.

  • Remaining exposure (NOTE N1): the delete snapshot now goes through GetMessage (about 8 statements: attachments, reactions, mentions, polls, forum, tags) under the row lock. If one of them failed deterministically for one message (a scan bug on a legacy row, say), that message could no longer be deleted with the normal route, where before the delete went through without detail. The mitigation already exists: bulk delete (moderation.go) still writes its entries after the commit and does not read this snapshot first. I found no such failing input.

(2) Longer transactions

What moved into a transaction:

  • delete: the snapshot (GetMessage, channel name, reply ids, edit versions), read after the row lock and the permission check, before the DELETE;
  • edit: the snapshot, the last message_edits row and the read of the stored message;
  • post and forum post: the read of the stored message (it was already loaded in the transaction when the channel had webhooks);
  • reaction: ReactionSummary;
  • conversation start: ListDMs;
  • event comment: the comment read;
  • RSVP: Store.Get (event, counts, signups);
  • webhook create: get;
  • wiki create: getPage;
  • register, report, event create, account delete: one INSERT into audit_log each.

There is no network or object-store call in any of these: all are SQL on the same transaction, and the object deletes (core.DeleteObjects) and every publish stay after the commit. Locks: the message row (FOR UPDATE for delete, FOR NO KEY UPDATE for edit and reaction) is now held for those extra reads, a few milliseconds at most. Registration holds the signup lock for one more INSERT.

Measured: post, edit and delete through the real routes, 40 rounds per writer, 1 writer and 16 concurrent writers (separate members, one channel), two runs per tree, no failures:

writersopbase p50 / p99fix p50 / p99
1post3.2 / 11.8, 2.3 / 4.8 ms2.3 / 5.8, 2.8 / 3.9 ms
1edit5.0 / 8.2, 3.9 / 9.7 ms3.2 / 6.3, 4.4 / 6.6 ms
1delete3.8 / 7.1, 3.2 / 6.2 ms2.5 / 4.3, 3.1 / 4.2 ms
16post9.9 / 38.1, 10.2 / 39.0 ms10.5 / 41.8, 9.9 / 34.4 ms
16edit14.0 / 24.7, 14.0 / 36.4 ms12.5 / 20.7, 11.9 / 20.6 ms
16delete10.5 / 18.4, 10.6 / 20.5 ms9.2 / 16.6, 8.8 / 16.3 ms

Post is unchanged within noise. Edit and delete are slightly faster, because the separate snapshot round trips before and after the change are gone.

(3) Changed existing tests: stricter or weaker

  • TestChatCancellationBetweenDatabaseStages, "cancellation after committed edit" (internal/chat/write_rollback_integration_test.go). Before, a cancellation at the commit had to answer context.Canceled with no event published. Now it must answer success, the stored text must be the edit's, and exactly one event must be published (len(recorder.Snapshot()) != 1 fails). So the event goes out exactly once, never zero times and never twice. This is stricter, and it is the H-L7-05 property itself.
  • TestAuditSnapshotOfAMissingMessageFails and TestAuditSnapshotFailsWhenTheChannelCannotBeRead (internal/chat/messages/audit_integration_test.go). The old tests asserted the old fail-open contract (nil snapshot, edit and delete go on without detail). The new ones assert core.ErrMessageNotFound and an error on a broken channel lookup, and keep the "no audit log, no snapshot, no error" case. This changes the contract in the direction the finding asks for, and is not weaker.
  • door_test.go. InsertStoredMessage is added to the pinned list of message doors, and the AST scan now also matches its callers. That list exists for exactly this: "a new caller fails it until it is reviewed and added". InsertStoredMessage applies rulesFrom(ctx) and automod.Of(p) exactly like InsertMessage (post.go:45-55), so the gate is kept, not weakened.
  • forum_errors_integration_test.go. A type change only (post.ID).
  • crash_points_integration_test.go. The knownPartial list only shrinks.

(4) Events and the outbox

Every WebSocket publish in the changed code runs after BeginFunc returned nil:

  • delete: tellAuthor, BroadcastChannel;
  • edit: BroadcastChannel after editMessage;
  • post: Service.Post announce, PublishStored;
  • reaction, conversation start, RSVP, event comment, register (announceRegistered), account delete (endDeletedAccount): after the store call.

The webhook outbox rows (recordMessageEvent, recordReactions) are written inside the transaction and roll back with it. automod's pending.Commit still runs after the commit. A rolled-back change therefore publishes nothing, and the deep sweep (every statement failed in turn) found no partial write outside the nine documented admin operations.

(5) Registration (RegisterAudited)

The entry is inserted after insertInvitedUser, in its transaction, only on success. Every refusal returns before it, as before: bad invite, raid lock, username taken, password rules. The old code also wrote the entry only after a successful registration, so status codes and error texts are unchanged, and the success path costs the same single INSERT, now inside the transaction. No new enumeration or timing signal.

(6) Removed exported API and changed signatures

Changes:

  • messages.StoredError removed;
  • forum.CreateForumPost now returns *core.Message;
  • dm.ListDMs and withDMMembers take a db.Querier;
  • added: NewDM.Load, NewReport.Audited, auth.RegisterAudited, InsertStoredMessage, PublishStored, Store.AnswerRSVP.

A grep for StoredError over the whole tree (both modules, all build tags) finds nothing. Both go vet ./... with and without -tags=integration pass, which covers every test file: the only build tags are integration and !integration. go vet in the loadtest/viewer module passes. The scheduled sender's StoredError branch (internal/later/scheduled_send.go) is gone, and that is correct: an error now means nothing was stored, so the retry is right.

(7) The documentation list against a grep of every after-commit write

I grepped every RecordOrLog( and every wrapper (AuditUser, auditRecord, wiki record) in the fixed tree, 32 direct sites plus the wrappers, and checked each one against "When an entry can be missing":

  • Listed and true:

    • bulk delete (moderation.go:200,202), pins.go:212, poll/handler.go:176;
    • forum: handler_forum.go:300, handler_forum_extras.go:120,177, handler_forum_tags.go:207;
    • Talks: talks_handler.go:102 (Talk settings), talks_live.go:177 (removal of a temporary Talk), server/audit_presence.go:67 (voice.join and voice.leave);
    • events: handler.go (edit, cancel), handler_cover.go (covers), handler_series.go (series), handler_roles.go, handler_templates.go, handler_comments.go:235 (comment removals);
    • wiki: handler_write.go (edit, move, delete, restore, lock), images.go:170;
    • roles/handler_self.go:92, channels/auto_delete.go:153, automod/hits.go:128 and raid.go:45, memberadmin/moderation_bans.go:273 (expired bans), media/avatar_reencode.go:147, slashcmd.go:103, edit_history.go:179;
    • the auth events (login, failed login, password change, reset mail, e-mail, second factor, sessions and logout_all, data export);
    • the middleware.

    The seven moved operations are indeed written by RecordTx.

  • Missing: S1 (SHOULD-FIX). user.talk_kick is written by RecordOrLog after the commit in internal/auth/memberadmin/users_admin_talk.go:109. That is POST /api/channels/{id}/voice/{userID}/disconnect, which the code comment says is "Outside the admin area, so the audit middleware does not see it". It is a moderation action, and the list does not name it ("Talks (voice.join and voice.leave, Talk settings, the removal of a temporary Talk)"). Add it.

  • Incomplete cause: S2 (SHOULD-FIX). The section names the causes as "a database error or restart, or the server dying in that moment". Every after-commit site passes the request context (r.Context(), or a ctx derived from it: middleware.go:130, all the handler sites above). net/http cancels that context when the client's connection closes, and pgx fails a statement on a cancelled context at once. So a client that disconnects right after the commit also loses the entry, and that is the one cause the actor controls. Either name it in the section, or (better, one line) make RecordOrLog write with context.WithoutCancel(ctx) under a short timeout, which removes this cause for all 30-odd remaining sites. I did not run a client-abort repro. The mechanism is net/http's documented behaviour plus pgx's, and I raised it in phase 2 (refute-H-L7-04-06.md) as a suspicion.

  • doc/architecture.md: the added paragraph is true. It names the typed message and the forum post as read in the transaction, and the edit, reaction, conversation, comment, RSVP and webhook paths.

Further notes

  • N1: the delete snapshot fails closed; see (1). Bulk delete is the fallback. No action needed.
  • N2: two message paths still read their answer after the commit through PublishMessage: forwarding (internal/chat/messages/forward.go:188) and uploads (internal/media/handler_upload.go:228). A database error in that moment still answers 500 for a stored message. They were not among the swept operations (the finding's "Not shown" covers them) and the docs do not claim otherwise. A follow-up candidate, not a blocker.
  • N3: mixed contexts in the edit. The edit's transaction runs on ctx from ctxFor, while auditEdit runs its statements on r.Context(). Both derive from the request, so a cancellation fails the whole transaction, consistently. Harmless today; one context would be tidier.
  • N4: duplicate reports. When a member reports a message they already reported, storeReport finds the open report and returns it, and Audited writes another report.create for it. The base did the same after the commit, so this is not a behaviour change.

Conflicts to expect with the lane 5 fixes

A trial git merge-tree of this branch against each lane 5 fix branch:

  • msch128/fix-h-l5-01: CONFLICT in internal/chat/messages/handler_messages.go (EditMessageWith, DeleteMessage) and internal/chat/messages/message_mutations.go (EditMessage, which here becomes the wrapper of the new editMessage with the audit hook and the read in the transaction, and DeleteMessageAs, which becomes the wrapper of deleteMessageAs with the hook after checkDeleteAuthorized). Merge by hand. Keep both the audit hook (it must run inside the transaction, before the commit) and lane 5's readstate recount. Keep any new publish after the commit.
  • msch128/fix-h-l5-04: clean (it touches insertMessageTx and ScreenText in post.go). The neighbouring hot spots in this branch are insertMessage, the new finishInsert, PublishStored (which now calls MarkReadAfterSend) and Service.Post in service.go (the fallback GetMessage after the commit is removed).
  • msch128/fix-h-l5-05: clean (thread_follow_sync.go).

Not checked

  • A real client abort timed against the commit (S2 rests on documented net/http and pgx behaviour).
  • Lock hold time under contention on one hot message (reactions from many members on the same message at once). The latency runs used separate messages.
  • The web client: no web files are changed.
  • The author's own suite logs were read only for their exit codes (all 0). Every result above is from my own runs.

Round 2: ecb083716 "fix: an audit entry written after the commit does not end with the request"

The delta is on top of dc8adbe77. It answers S1 and S2 from round 1. Read in full: internal/audit/audit.go, internal/audit/middleware.go, internal/audit/fake_test.go and doc/moderation.md.

Verdict H-L7-04: MERGE. S1 and S2 are resolved; one NOTE (R2-N1) is left for a follow-up.

What changed

afterCommit(ctx) is context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second). RecordOrLog writes through it. The admin middleware builds one such context after the handler and uses it for the "after" snapshot (finalizeEntry) and for the entry. The documentation names "removing a member from one Talk" (S1) and says a disconnect no longer loses the entry; what is left is a crash or a database failure (S2).

Checks

  • No cancelled context left on the write path. Inside RecordOrLog everything uses the detached context:

    • Record → RecordTx → l.Actor(ctx), which is auditActor and reads context values only (internal/server/hooks.go:23);
    • the pool acquire and the INSERT (q.Exec(ctx, ...) on the pool).

    In the middleware, the after-snapshot (addStateChange/addCreatedState → Snapshot(ctx, l.DB, ...)) now gets the detached context too. The before-snapshot still uses the request context, which is correct: it runs before the change. A cancellation there means the handler most likely fails as well, and then no entry is due.

  • Pile-up under a slow database. The write runs synchronously in the caller's goroutine. No goroutine is started, so the number of writes at once is bounded by the requests in flight (plus the few background callers), and each waits at most 5 s, the pool acquire included. Before, a disconnected request ended at once. Now its goroutine lives up to 5 s longer, and only when the database is slow. The background callers that passed context.Background() (automod/hits.go:128, raid.go:45) had no bound at all before and now have 5 s, which is an improvement. The middleware's nested afterCommit inside RecordOrLog takes the earlier deadline, so the whole after-handler work stays within 5 s.

  • Shutdown. These writes are not background jobs, so Drain need not track them. They run inside the handler, and http.Server.Shutdown waits for handlers for 15 s, longer than the 5 s cap. Only a handler that already outlived the 15 s (H-L7-08's case) can still write when main's defers run pool.Close(). Then either the pgxpool acquire on the closed pool fails at once (logged as "audit log write failed"), or a write already holding a connection delays pool.Close by at most the 5 s cap. Nothing hangs and nothing panics.

  • The test fails without the change. With audit.go and middleware.go from dc8adbe77 and the new test kept: fake_test.go:109: RecordOrLog with a cancelled request context wrote 0 entries, want 1, FAIL. With ecb083716: PASS.

  • A real pool, once. In a reviewer-only integration probe (not committed), RecordOrLog with an already-cancelled context against the real PostgreSQL pool wrote 1 entry on ecb083716 and 0 entries with the dc8adbe77 audit code.

  • Gates. All exit 0 or ok:

    • go vet (internal/audit; with -tags=integration internal/audit and internal/server), gofmt;
    • unit tests of internal/audit and internal/repo;
    • scripts/complexity-check.sh, make docs-check, scripts/check-english-only.mjs;
    • go test -tags=integration -run Audit over internal/server, internal/audit and internal/chat/...

Note

  • R2-N1 (NOTE, follow-up). The new sentence in the documentation, "a client that disconnects in that moment still leaves its entry", is true of the write itself. It is not true of handlers that do a post-commit read on the request context first and skip the entry when that read fails. Example: internal/chat/talks/talks_handler.go:89-105 (Talk settings, channel.update). After EditChannel has committed, FillTalkFields(r.Context()) returning an error leaves the handler before the entry, and so does IsDMTalk(r.Context()) ("talk update not audited"). A disconnect there still loses the entry, and the client is answered 500 for a committed change (the H-L7-05 shape). Other sites only lose detail on such a failure, which I checked at poll.go auditEnd and users_admin_talk.go:105: they still write the entry. Low moderation value. A follow-up should either move that read before the commit or use afterCommit for it, or the documentation should say "the write itself". Not blocking.

Something wrong on this page? Tell us.