Fix review of H-L7-01, H-L7-02, H-L7-03 (lane 7, phase 3)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: fix reviewer (independent; I wrote neither the findings nor the fixes). Read-only: nothing in the repository was changed, nothing pushed. Each fix is one commit on the lane 7 test commit 85dc7239d:

FixBranchCommit
H-L7-01msch128/fix-h-l7-013525e3398 fix: issue account mail links under one lock per account and purpose
H-L7-02msch128/fix-h-l7-02d1663129f fix: seed the Owner once when servers start together on an empty database
H-L7-03msch128/fix-h-l7-03c50688c02 fix: count and create event role templates under one lock

Inputs: findings H-L7-01..03, phase 2 reports repro-H-L7-01-03.md and refute-H-L7-01-03.md. Method: one temporary worktree per branch (removed afterwards), the built web app copied in so that internal/server and cmd/server compile, Docker for the integration tests, nice -n 10, -p 1, one heavy job at a time. My own probes were temporary _test.go files in the worktrees, never committed.

Verdicts

FixVerdict
H-L7-01MERGE AFTER FIXES: one SHOULD-FIX in the test (an occasional race-detector failure, from the lane 7 test, not from the fix)
H-L7-02MERGE
H-L7-03MERGE

What the fix does

IssueAccountToken (internal/auth/account_token.go:78) now prunes long-expired rows on the pool, then runs replaceToken (:99) in one transaction: pg_advisory_xact_lock(hashtextextended('account_token:' || user_id || ':' || purpose, 0)), the hourly count (recentTokens), retire, insert. Past TokensPerHour it stores nothing and returns the new auth.ErrTooManyLinks. sendReset (password_reset.go:95) ends silently on it, sendVerification (email_handler.go:227) answers 429, the same as for the early count. The signature changes from db.Querier to *db.Pool (an internal Go package, not covered by doc/api-stability.md; every caller already passed the pool).

Cause removed?

Yes, for every variant the finding and the refuter name. Both purposes go through the one function. No other code inserts into account_tokens (I grepped for IssueAccountToken, IssuedTooOften, ErrTooManyLinks and INTO account_tokens). Under read committed each statement after the lock takes a fresh snapshot, so the count sees what the previous holder committed. Probes that are not in the tests, run on the fix and on the base commit:

ProbeFixBase
32 callers at once, mixing 2 accounts x 2 purposes, 10 rounds: per (account, purpose) exactly 1 live link and exactly 5 links in the hour; 20 successes in all, the rest ErrTooManyLinks0 of 10 rounds bad10 of 10 bad (8 live, 8 in the hour per pair)
Budget partly spent (4 sequential links), then 8 at once: exactly 1 succeeds0 of 10 bad10 of 10 bad (8 succeed)
Lock scope: a transaction holds the key for (member, reset); issues for (other account, reset) and (member, verify)finish in 4 ms; the same key waits until the holder ends(no lock)
Reset form, 300 known and 300 unknown addresses, handler call timed (the work runs after it)both 202, empty body; p50 1.7 / 1.5 µsboth 202, empty body; p50 1.9 / 1.8 µs

Lock key and collisions

Every advisory lock in the code base takes the single-bigint form, so they all share one key space:

  • Constants: migrations 0x6d6e656d61, backup 0x6d6e656d6162 (shared/exclusive), channel layout 0x6d6e6c6179, slugs 7301265904, emoji 0x656d6f6a69, permissions 0x6d6e7065726d. All are above 2^31.
  • hashtext (32-bit, sign-extended): 'wiki_images', 'mnema.wiki', 'staged:' || user, plus the new 'mnema.owner_seed' (H-L7-02) and 'mnema.event_role_templates' (H-L7-03).
  • hashtextextended (64-bit): 'export:' || user, 'push:' || user, poll message || user, plus the new 'account_token:' || user || ':' || purpose.

The new key is 64-bit and its text has a prefix and separators, so it matches another key only by a 64-bit hash collision. A collision would only make two short transactions wait for each other. The token transaction takes no second advisory lock, so a collision cannot close a wait cycle.

Can a flood starve the pool? (measured)

Production pool: DB_MAX_CONNS 25, DB_STATEMENT_TIMEOUT 15 s. Anonymous reset requests reach IssueAccountToken only through background (account_mail.go:25 mailJobs = 8: a full queue drops the job) and only after resetPerAccount (5 per hour per account, in memory). So per process at most 8 issues run at a time, and at most 5 per hour queue on one victim's key. Before the fix the same calls ran unlocked, but they still used one connection each. Measured with a 25-connection, 15 s pool, with a probe query (the sign-in user lookup) every 2 ms during the burst:

Burst on one accountFixBase
8 callers (the route's bound)11 ms; 5 issued, 3 ErrTooManyLinks; probe p50 0.21 ms, max 0.61 ms3 ms; 8 issued; probe max 0.90 ms
200 callers (store level, more than the route can produce)66 ms; 5 issued, 195 refused; probe p50 0.22 ms, max 54 ms; no errors36 ms; 200 issued; probe max 23 ms

Each waiter holds one connection and needs no second one, so a full pool queues and does not deadlock. Waits are milliseconds, far below the statement timeout. Sign-in for everybody else is not starved. The signed-in verify path is gated by ConfirmIdentity (password check) or resendPerUser (1 per minute), so it cannot produce a larger burst.

The new error is handled by every caller; the reset path stays silent

  • sendReset: returns on ErrTooManyLinks with no log line, the same as on the early count. The response is written before the background job and was not changed. Code and body are the same for known and unknown addresses, before and after (table above). The small p90 difference between known and unknown (13 µs against 4 µs) is the same on the base commit: CPU taken by the background job, not something the fix adds.
  • sendVerification: maps it to httpx.ErrRateLimited(). Through setEmail that is a mailRefused (confirmation_sent: false), through resend a 429, the same as the existing early count. A caller that issues links one after another sees no change: when the early count passes, the locked count passes too.
  • Tests that call IssueAccountToken directly: email_mail_integration_test.go:46, password_reset_integration_test.go:43 and the races file all pass. The one that issued more than 5 in an hour (TestParallelUsesOfOneMailLinkLetExactlyOneWin) now clears the tokens each round, which is correct.

Prune outside the lock

The DELETE only removes rows that expired more than a day ago, so it cannot change the hourly count of rows created in the last hour. It is one autocommit statement before the transaction, as before. With the fix there is at most one unused row per (account, purpose). The retire UPDATE therefore locks at most one row, and a prune running at the same time cannot form a two-row wait cycle with it.

SECURITY.md

Only the one sentence (SECURITY.md:333) changed (git diff --stat: 1 line). It is now true. The later "per account (5 mails per hour, counted in the database so it survives restarts …)" was not true under concurrency after a restart; it is true now as well.

Findings

SHOULD-FIX 1: TestParallelResetRequestsLeaveOnlyTheNewestLinkWorking fails now and then under -race (an Add/Wait misuse of a WaitGroup in the test). internal/auth/authn/reset_race_integration_test.go:68-73. Each caller calls h.DrainMail() (wg.Wait) while the other callers of the same round are still in requestPasswordReset, whose background calls wg.Add(1) (password_reset.go:68, account_mail.go). An Add that moves the counter up from zero while a Wait is running breaks the sync.WaitGroup contract, and the race detector reports it. Evidence: in my first full -race run of ./internal/auth/... the test failed with "race detected during execution of test". The report: write by one race caller (racetest.go:109), previous read at password_reset.go:68 in another caller of the same round. I then ran 12 isolated runs and 4 full authn package runs with -race, all clean, so the rate is about 1 in 17 runs here. The test code is the same on 85dc7239d (the lane 7 test commit), so the fix did not cause it. It matters now because the test stops being an expected failure and becomes a normal test in make test-integration (which uses -race) on the branch that will be merged. Suggested fix: remove h.DrainMail() from the caller and keep the one in the check (:76). That check already waits for every mail of the round before it counts.

NOTE 1: NOW() inside the transaction is the transaction start, before the lock wait. A waiter's count window and its created_at/expires_at are therefore early by the wait (milliseconds). This has no practical effect.

NOTE 2 (outside the fix): Handler.Drain at shutdown has the same Add-during-Wait pattern against requests that are still arriving. It is pre-existing and only shows in -race builds. I did not investigate it further.

Gates

gofmt clean; go vet with and without -tags=integration on internal/auth/..., internal/server, cmd/server; make complexity-check 0; make staticcheck 0; internal/repo tests ok; English check ok. account_token.go has 172 lines. Commit type fix:, no !, no attribution lines. No allowlist entry, no raised limit, no skipped test.


H-L7-02: Owner seed under an advisory lock

What the fix does

EnsureAdminUser keeps its unlocked checkExistingAdmin as the fast path. Only when no Owner exists does createAdminAccount (admin_seed.go:102) hash the password, then run one transaction: pg_advisory_xact_lock(hashtext('mnema.owner_seed')), re-check hasAdminUser, check the name, insert. A server that finds an Owner inside the lock logs "an administrator was created by another server starting at the same time" and returns nil. The name check moved inside the lock, so "taken by the regular user X" is true again. No migration and no unique index; the commit message explains why (an index would fail to build on a database that already has two Owners from this race).

Cause removed? Probes

Recorded test (both subtests) passes 5 of 5 runs; reverted, it fails 10 of 10 rounds in each subtest ("taken by a regular user" and "2 owners"). Own probes, on the fix and on the base:

ProbeFixBase
Upgrade path: an Owner exists while another session holds the seed lockreturns nil in 3 ms (no lock taken)3 ms
Two development starts with an empty password, 10 rounds: exactly one generated password logged, by the server that created the Owner, and it signs in; one "another server" line; both starts nil0 of 10 bad10 of 10 bad (loser fails with the false message)
Names that differ only in case (Owner, owner)0 of 10 bad, one Owner10 of 10 bad
Name of an existing regular user, no OwnerADMIN_USERNAME "Member" is taken by the regular user "member" (true)same
The seeding server dies inside the transaction (its backend terminated after it took the lock and inserted an uncommitted Owner)the waiter gets the lock, seeds its own Owner, returns nil after 0.44 s; one Owner (the waiter's)(no lock)
The seeding server hangs inside the transaction (2 s statement timeout)the waiter fails with lock admin seed: … statement timeout (57014) after 2.0 s, no Owner(no lock)

The generated password is logged only after created and the commit, so a server that loses never logs one (admin_seed.go:122-127; probe above). A server that dies releases the lock and rolls back its insert when its connection closes.

Lock key

hashtext('mnema.owner_seed') = 1629348251. It differs from the other static hashtext keys (mnema.event_role_templates -51764638, wiki_images 294820243, mnema.wiki -375449778). It cannot equal any constant key, because those are all above 2^31. It can collide only with a per-user 'staged:' || user key, with probability 2^-32 per user, and only at the first start. At that point no user exists, so no upload staging can hold such a key. ADMIN_USERNAME is read nowhere else at run time (grep), so a loser with a different name has no other effect.

Findings

NOTE 3: When the seeding server hangs (does not die), a waiter fails its start after DB_STATEMENT_TIMEOUT (15 s), and the holder is cut off only by the 1-minute idle_in_transaction_session_timeout. That can mean a few restarts in a row. This is only an artificial hang (the real holder runs three short statements) and the restart policy covers it.

NOTE 4 (unchanged behaviour): A losing server whose seed settings are invalid (for example an empty ADMIN_INITIAL_PASSWORD in production) still fails before the lock with that error, where a later start would only warn. This is the same as before the fix, and the docs sentence ("start normally, as any later start does") is true for valid settings.

Docs: the doc/roles-and-permissions.md sentence matches. Gates as for 01, all green; admin_seed.go has 157 lines.


H-L7-03: role templates counted and created under a lock

What the fix does

CreateTemplate (store_templates.go:75) runs the count and the insert in inTx after pg_advisory_xact_lock(hashtext('mnema.event_role_templates')). The unique index on the name still maps to the same 409 "a template called X exists". The cap error and its 409 are unchanged. No other code path inserts templates (only migration 0086 seeds the presets). Update and delete cannot raise the count. The production Store.DB is the pool (router_build_services.go:114), which inTx needs. Other paths of the store already depend on that.

Probes

Recorded test passes 5 of 5 runs; reverted, it fails 10 of 10 rounds (52 to 57 templates, 3 to 8 creates). Own probes, on the fix and on the base:

ProbeFixBase
8 creates with the same name at 30 templates: 1 created, 7 x 409 "a template called Same name exists"0 of 10 bad0 of 10 bad (unchanged)
At 50: 3 deletes and 8 creates at once; final count <= 50 and equal to 47 + creates0 of 10 bad0 of 10 bad (not discriminating: the deletes make room; shows no regression)
Through the handler (createTemplate as the Owner), 8 at once at 49: one 201, seven 4090 of 10 bad10 of 10 bad (8 created, 57 templates)

Lock key: hashtext('mnema.event_role_templates') = -51764638. It differs from every other static key and cannot equal a constant. A collision with a per-user staged: key (probability 2^-32 per user) would only make two short transactions wait for each other. Neither transaction takes a second advisory lock, so no wait cycle is possible.

No findings. Gates as for 01, all green; store_templates.go has 160 lines.


What I ran

All with DOCKER_HOST set to the local OrbStack socket, nice -n 10, -p 1.

  • Per branch: gofmt; go vet and go vet -tags=integration (internal/auth/..., internal/communityevents, internal/server, cmd/server); make complexity-check; make staticcheck; go test ./internal/repo/; node scripts/check-english-only.mjs. All exit 0.
  • Recorded race tests 5 times per fix (01: both store tests and the route test; 02; 03). All pass.
  • Revert check per fix: production files reset to 85dc7239d, tests kept. 01 needed a one-line ErrTooManyLinks stub so the tests build. Every recorded test fails in every round.
  • Full integration suites with -race -count=1:
    • 01: internal/auth/... and cmd/server: all ok except the race-detector failure in SHOULD-FIX 1 (reruns clean: 12 isolated runs and 4 full authn runs). internal/server: ok (489 s).
    • 02: internal/auth/..., cmd/server, internal/server: all ok, no data race.
    • 03: internal/communityevents (unit and integration), cmd/server, internal/server: all ok, no data race.
    • No "can't assign requested address" failures occurred.
  • Unit tests go test -race ./internal/auth/... on 01 and 02: ok.
  • The probes in the tables above, on each fix and on 85dc7239d.

What I did not check

  • More than one real server process: H-L7-02 was probed with two seeders in one process and a terminated backend, not with two binaries.
  • The SFU, web and e2e suites (the fixes touch none of them). make openapi-check was not run (no annotation changed).
  • Production Handler.Drain beyond NOTE 2.
  • I did not mutate the fixes beyond the revert.

Something wrong on this page? Tell us.