H-L7-08: phase 2 (refutation) and blocking fix review

From mnema-talk v3.3.13, rendered on this site.

Reviewer: independent; wrote neither the finding, the tests nor the fix. Read-only. Tests: msch128/hunt-c-l7-a at 8a5fbccbf. Fix: msch128/fix-h-l7-08 at 49c4a137a (one commit on top of 8a5fbccbf, base origin/main f58bc3fc6, 3.0.9). Go 1.27.4. Each branch in its own temporary worktree, removed afterwards. One heavy job at a time, nice -n 10, -p 1; load was 4.6-5.8.

Phase 2

Reproduction (base 8a5fbccbf)

MNEMA_DRAIN_HAMMER=1 go test -p 1 -count=1 -run 'TestKnownGap|TestDrainHammer' -v ./internal/auth/authn ./internal/updaterd

REPRODUCED. All four TestKnownGap* tests pass, so the gap is present: a mail job is accepted after Drain returned and after Drain timed out, a tracked job is accepted after Drain, and a swap is accepted after WaitIdle returned. TestDrainHammer logs child: err=exit status 2 misuse-panic=true race=false after 0.26 s.

Attempts to defeat it

  • Reachable in production, server side. cmd/server/main.go:128 defers pool.Close(); :172 defers router.Close(), so Router.Close runs first (LIFO). internal/server/run.go:47-50 calls srv.Shutdown with a 15 s timeout and returns, and main's defers then run Router.Close (internal/server/server.go:100-108). That calls Hub.Close, cancels the lifetime and then calls a.Drain(jobsStopWait). http.Server.Shutdown returns at its deadline without stopping handlers that are still running. So a handler in flight past 15 s keeps running while Drain runs and after it returns, and can still reach background (password_reset.go:67, before any account lookup; two_factor_alert.go:18) or TrackJob (account_export.go, after claimExport). The window is real but needs a handler slower than 15 s at the moment of a shutdown. In practice that means database trouble: pool acquisition or the export's per-account advisory lock blocking.
  • Is the panic reachable, or only the acceptance? The misuse panic needs Wait to be sleeping, so jobs were in flight when Drain began, then the last job finishing and a new Add from zero landing in the nanoseconds before Wait returns. Wait runs under safego.Crash (account_mail.go Drain), so a panic ends the process during shutdown: exit 2 instead of a clean exit, and the defers that follow (pool.Close, voice and relay close) are skipped. In production the panic is a theoretical tail. The acceptance is the reachable part: a job that runs against a pool that is closing, or that is cut off when the process exits.
  • Reachable in production, updater (stronger than the finding says). internal/updaterd/run.go: hs.Shutdown has 5 s, but verifyTimeout is 90 s (server.go:48). A POST /v1/update still verifying at SIGTERM outlives Shutdown. On the base, WaitIdle then sees zero swaps, returns true, and run returns 0, so the process exits. If that handler reaches background in that span, a swap starts and is killed at an arbitrary point. That is the very state the grace period is there to prevent: "the local tag may point at the new image while the app was not recreated" (run.go:165). Only the Owner triggers an update, so this needs special rights.
  • Documented or intended? No. The Drain comment promised that shutdown "does not close the database under" the jobs, and no test asserted acceptance after Drain.
  • Duplicate? No. It is related to the H-L7-01 reviewer's race observation, but this is the first finding about the drain itself.
  • Severity. S4 fits: unusual conditions (shutdown plus a handler slower than the timeout, or an Owner-triggered update at the same moment as an updater stop). Nothing is exposed and no data is corrupted beyond a lost mail or an export marked failed. The updater case can leave an inconsistent swap, but it needs the Owner and a coincidence.

Phase 2 verdict: STANDS (S4). The updater variant is somewhat more reachable than the finding states (90 s verification against a 5 s Shutdown). The panic is a tail risk; the acceptance after Drain is the real defect.

Fix review: 49c4a137a "fix: refuse background jobs once shutdown drain began (H-L7-08)"

What it does

  • recovery (internal/auth/authn/account_mail.go) gains mu and draining. enter() takes mu, refuses when draining, else calls wg.Add(1). background takes a queue slot, calls enter, and on refusal releases the slot and logs "server shutting down, dropping account mail". TrackJob now returns (done, ok). Drain sets draining under mu, releases mu, then waits.
  • The export (internal/auth/account/account_export.go): buildExportInBackground returns a *httpx.APIError. On refusal it frees the build slot, calls failExport (status 'failed', which does not count toward the daily limit) and answers 503 "server shutting down" with Retry-After.
  • updaterd (internal/updaterd/server.go): admit() does the check and the Add under s.mu. handleUpdate calls it after verification and before the 202; a refusal answers 503, and the deferred release() clears busy because handedOver stays false. WaitIdle sets draining first.

Checks

  • Cause removed. Every wg.Add / swaps.Add in these two types is now under the mutex, after the flag check. The flag is set under the same mutex before Wait. So any Add happens before Drain's unlock, which happens before Wait (no concurrent Add from zero), and none follows it. A grep of the whole tree for TrackJob, .background(, WaitIdle, DrainMail and .Drain( finds these production callers only: password_reset.go:67, two_factor_alert.go:18, account_export.go:123, updaterd/server.go:146/153, run.go:164, server.go:107. All go through the guard. DrainMail is called only from tests, and its comment now says it must not run concurrently with requests.
  • Lock order / blocking under the lock. recovery.mu is a leaf lock: held only for the flag and an Add, never across a channel send, the goroutine start or Wait. In updaterd, s.mu is shared by acquire, release, admit and WaitIdle, but none of them nests another lock or blocks while holding it; the swap's release() takes it briefly from the swap goroutine. No deadlock path found.
  • A job that starts a job during Drain (my input 1). No current job starts another: mail jobs send mail, and the export build does not call background or TrackJob. If one did, the inner start would be refused and dropped, not deadlocked, because the mutex is not held while a job runs or while Wait waits.
  • Timed-out Drain (my input 2). draining stays true. TestMailJobIsRefusedAfterDrainTimedOut covers a mail job; by reading, a second Drain after the stuck job ends returns true, and TrackJob after a timed-out Drain is refused through the same enter().
  • Reset 202 indistinguishable (my input 3). background is called before any account lookup (password_reset.go:65-68; the lookup is inside sendReset), so a refusal does not depend on the account. The answer is always 202. The refused path (release slot, log) costs about the same as an accepted one (Add, goroutine start), and both are nanoseconds next to the request. Nothing can be enumerated.
  • Export on a closed pool. The refusal happens only after Drain began. Router.Close runs before pool.Close (defer order above), so failExport normally works. After pool.Close, pgxpool rejects the acquire at once (no hang), and the existing sweep marks a 'pending' export failed after 30 minutes (sweepExports, account_export.go:335). The new comment says exactly this. Until then, a 'pending' row counts toward the daily limit (status <> 'failed'); that only happens when a refused request arrives after the pool closed, a few milliseconds before exit.
  • Public API. POST /api/users/me/export already documented a 503 UNAVAILABLE with Retry-After. The fix adds "or the server is shutting down" to the swag annotation and api/openapi.json (both changed, consistently). The text "server shutting down" is an existing, known message (internal/repo/testdata/error_messages.golden.json:403, also internal/ws/lifecycle.go:112). It is not a contract change under doc/api-stability.md, which lists no export status semantics. /v1/update is the internal sidecar API and has no entry in doc/ or OpenAPI. The server's client (internal/update/updater.go:141-142) maps an unknown 503 to "the updater answered HTTP 503", which is acceptable.
  • Revert checks (run).
    • updaterd flag check neutralised (_ = s.draining, Add kept under the mutex): TestNoSwapIsAdmittedAfterWaitIdleReturned fails ("a swap was admitted after WaitIdle"), and TestUpdateRequestAfterWaitIdleIsRefusedWith503 fails ("status = 202, want 503").
    • authn flag check neutralised, Add kept under the mutex: TestDrainHammer fails (misuse-panic=true race=false, 0.20 s), TestMailJobIsRefusedAfterDrainReturned fails, and TestMailJobIsRefusedAfterDrainTimedOut fails.
    • The same mutex-only variant under -race: misuse-panic=true race=true, "WARNING: DATA RACE". So the refusal, not the mutex, removes the panic. The mutex alone does not remove the data race either: Wait is not under the mutex, so a mutex-protected Add from zero still has no happens-before edge to a concurrent Wait. Only the flag, set under the mutex before Wait and checked before every Add, creates that order.
  • Hammer under -race on the fix (run). MNEMA_DRAIN_HAMMER=1 go test -race -p 1 -count=1 -run 'TestDrainHammer|Drain|Refused|WaitIdle|Admitted' -v ./internal/auth/authn ./internal/updaterd passes: TestDrainHammer took 13.27 s (10 s of hammering in the child), and every new authn and updaterd test passes.
  • Other runs. go test -p 1 -count=1 ./internal/auth/authn ./internal/updaterd ./internal/auth/account passes. The integration tests TestExportRequestWithAllSlotsBusyIsRefusedAndNotRecorded, TestExportRequestAfterDrainIsRefusedAndMarkedFailed and TestShutdownCancelsExportsInFlightAndWaitsForThem pass (Docker). go vet passes with and without -tags=integration on the touched packages, gofmt is clean, scripts/complexity-check.sh exits 0, and the internal/repo tests (file and function length) pass. File sizes: account_mail.go 190, updaterd/server.go 321, account_export.go 359 lines; the hammer test is 80 lines.
  • Gates and style. No allowlist entry and no limit raised. Commit type fix:, no !, no attribution lines. English only. The known-gap tests became plain regression tests with inverted asserts. The hammer stays gated by MNEMA_DRAIN_HAMMER=1 and is a deep probe, not a smoke test.
  • Behaviour beyond the finding. Only the shutdown path changes. Outside shutdown, enter() adds one uncontended mutex per mail or export job, which is negligible (not a hot path).

Findings

  • NOTE: updaterd has no hammer. Its correctness rests on the same argument plus the two unit tests. The revert check shows the unit tests catch the refusal, but not a regression back to an Add outside the mutex. That is acceptable here: a swap is serialised by busy and starts at most once per request.
  • NOTE: on the server side, a 503 from the updater shows up as the generic "the updater answered HTTP 503". A dedicated ErrUpdaterShuttingDown would read better in the System tab. Optional.
  • NOTE: the hammer is not wired into any job. It should be: add it to the nightly sweep (the sim-sweep workflow on the hunt branches) as one step, MNEMA_DRAIN_HAMMER=1 go test -race -p 1 -count=1 -run '^TestDrainHammer$' ./internal/auth/authn, about 15 s with -race (10 s of hammering plus build). The window is nanoseconds wide, so the normal CI tests cannot catch a regression to an unguarded Add. Only the hammer does, and the race detector makes it reliable (the race showed within 3 s in the revert check).

No BLOCKER, no SHOULD-FIX.

Verdict: MERGE.

Not checked

  • make openapi-check was not run. The annotation and the JSON changed by hand in the same way, and I did not regenerate the JSON to confirm byte equality.
  • No real server shutdown with a slow handler was run, only the unit and integration tests.
  • staticcheck, ESLint and vue-tsc were not run (no web files touched).

Something wrong on this page? Tell us.