Fix review of H-L7-01, H-L7-02, H-L7-03 (lane 7, phase 3)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: fix reviewer (independent; I wrote neither the findings nor the fixes). Read-only: nothing in the repository was changed, nothing pushed. Each fix is one commit on the lane 7 test commit 85dc7239d:
| Fix | Branch | Commit |
|---|---|---|
| H-L7-01 | msch128/fix-h-l7-01 | 3525e3398 fix: issue account mail links under one lock per account and purpose |
| H-L7-02 | msch128/fix-h-l7-02 | d1663129f fix: seed the Owner once when servers start together on an empty database |
| H-L7-03 | msch128/fix-h-l7-03 | c50688c02 fix: count and create event role templates under one lock |
Inputs: findings H-L7-01..03, phase 2 reports repro-H-L7-01-03.md and refute-H-L7-01-03.md.
Method: one temporary worktree per branch (removed afterwards), the built web app copied in so that
internal/server and cmd/server compile, Docker for the integration tests, nice -n 10, -p 1, one
heavy job at a time. My own probes were temporary _test.go files in the worktrees, never committed.
Verdicts
| Fix | Verdict |
|---|---|
| H-L7-01 | MERGE AFTER FIXES: one SHOULD-FIX in the test (an occasional race-detector failure, from the lane 7 test, not from the fix) |
| H-L7-02 | MERGE |
| H-L7-03 | MERGE |
H-L7-01: account mail links under one lock per account and purpose
What the fix does
IssueAccountToken (internal/auth/account_token.go:78) now prunes long-expired rows on the pool, then
runs replaceToken (:99) in one transaction: pg_advisory_xact_lock(hashtextextended('account_token:' || user_id || ':' || purpose, 0)), the hourly count (recentTokens), retire, insert. Past TokensPerHour
it stores nothing and returns the new auth.ErrTooManyLinks. sendReset (password_reset.go:95) ends
silently on it, sendVerification (email_handler.go:227) answers 429, the same as for the early count.
The signature changes from db.Querier to *db.Pool (an internal Go package, not covered by
doc/api-stability.md; every caller already passed the pool).
Cause removed?
Yes, for every variant the finding and the refuter name. Both purposes go through the one function. No
other code inserts into account_tokens (I grepped for IssueAccountToken, IssuedTooOften,
ErrTooManyLinks and INTO account_tokens). Under read committed each statement after the lock takes
a fresh snapshot, so the count sees what the previous holder committed. Probes that are not in the tests,
run on the fix and on the base commit:
| Probe | Fix | Base |
|---|---|---|
32 callers at once, mixing 2 accounts x 2 purposes, 10 rounds: per (account, purpose) exactly 1 live link and exactly 5 links in the hour; 20 successes in all, the rest ErrTooManyLinks | 0 of 10 rounds bad | 10 of 10 bad (8 live, 8 in the hour per pair) |
| Budget partly spent (4 sequential links), then 8 at once: exactly 1 succeeds | 0 of 10 bad | 10 of 10 bad (8 succeed) |
| Lock scope: a transaction holds the key for (member, reset); issues for (other account, reset) and (member, verify) | finish in 4 ms; the same key waits until the holder ends | (no lock) |
| Reset form, 300 known and 300 unknown addresses, handler call timed (the work runs after it) | both 202, empty body; p50 1.7 / 1.5 µs | both 202, empty body; p50 1.9 / 1.8 µs |
Lock key and collisions
Every advisory lock in the code base takes the single-bigint form, so they all share one key space:
- Constants: migrations
0x6d6e656d61, backup0x6d6e656d6162(shared/exclusive), channel layout0x6d6e6c6179, slugs7301265904, emoji0x656d6f6a69, permissions0x6d6e7065726d. All are above 2^31. hashtext(32-bit, sign-extended):'wiki_images','mnema.wiki','staged:' || user, plus the new'mnema.owner_seed'(H-L7-02) and'mnema.event_role_templates'(H-L7-03).hashtextextended(64-bit):'export:' || user,'push:' || user, pollmessage || user, plus the new'account_token:' || user || ':' || purpose.
The new key is 64-bit and its text has a prefix and separators, so it matches another key only by a 64-bit hash collision. A collision would only make two short transactions wait for each other. The token transaction takes no second advisory lock, so a collision cannot close a wait cycle.
Can a flood starve the pool? (measured)
Production pool: DB_MAX_CONNS 25, DB_STATEMENT_TIMEOUT 15 s. Anonymous reset requests reach
IssueAccountToken only through background (account_mail.go:25 mailJobs = 8: a full queue drops the
job) and only after resetPerAccount (5 per hour per account, in memory). So per process at most 8
issues run at a time, and at most 5 per hour queue on one victim's key. Before the fix the same calls ran
unlocked, but they still used one connection each. Measured with a 25-connection, 15 s pool, with a probe
query (the sign-in user lookup) every 2 ms during the burst:
| Burst on one account | Fix | Base |
|---|---|---|
| 8 callers (the route's bound) | 11 ms; 5 issued, 3 ErrTooManyLinks; probe p50 0.21 ms, max 0.61 ms | 3 ms; 8 issued; probe max 0.90 ms |
| 200 callers (store level, more than the route can produce) | 66 ms; 5 issued, 195 refused; probe p50 0.22 ms, max 54 ms; no errors | 36 ms; 200 issued; probe max 23 ms |
Each waiter holds one connection and needs no second one, so a full pool queues and does not deadlock.
Waits are milliseconds, far below the statement timeout. Sign-in for everybody else is not starved. The
signed-in verify path is gated by ConfirmIdentity (password check) or resendPerUser (1 per minute), so
it cannot produce a larger burst.
The new error is handled by every caller; the reset path stays silent
sendReset: returns onErrTooManyLinkswith no log line, the same as on the early count. The response is written before the background job and was not changed. Code and body are the same for known and unknown addresses, before and after (table above). The small p90 difference between known and unknown (13 µs against 4 µs) is the same on the base commit: CPU taken by the background job, not something the fix adds.sendVerification: maps it tohttpx.ErrRateLimited(). ThroughsetEmailthat is amailRefused(confirmation_sent: false), throughresenda 429, the same as the existing early count. A caller that issues links one after another sees no change: when the early count passes, the locked count passes too.- Tests that call
IssueAccountTokendirectly:email_mail_integration_test.go:46,password_reset_integration_test.go:43and the races file all pass. The one that issued more than 5 in an hour (TestParallelUsesOfOneMailLinkLetExactlyOneWin) now clears the tokens each round, which is correct.
Prune outside the lock
The DELETE only removes rows that expired more than a day ago, so it cannot change the hourly count of
rows created in the last hour. It is one autocommit statement before the transaction, as before. With the
fix there is at most one unused row per (account, purpose). The retire UPDATE therefore locks at most
one row, and a prune running at the same time cannot form a two-row wait cycle with it.
SECURITY.md
Only the one sentence (SECURITY.md:333) changed (git diff --stat: 1 line). It is now true. The later
"per account (5 mails per hour, counted in the database so it survives restarts …)" was not true under
concurrency after a restart; it is true now as well.
Findings
SHOULD-FIX 1: TestParallelResetRequestsLeaveOnlyTheNewestLinkWorking fails now and then under -race
(an Add/Wait misuse of a WaitGroup in the test).
internal/auth/authn/reset_race_integration_test.go:68-73. Each caller calls h.DrainMail()
(wg.Wait) while the other callers of the same round are still in requestPasswordReset, whose
background calls wg.Add(1) (password_reset.go:68, account_mail.go). An Add that moves the counter
up from zero while a Wait is running breaks the sync.WaitGroup contract, and the race detector
reports it.
Evidence: in my first full -race run of ./internal/auth/... the test failed with "race detected
during execution of test". The report: write by one race caller (racetest.go:109), previous read at
password_reset.go:68 in another caller of the same round. I then ran 12 isolated runs and 4 full
authn package runs with -race, all clean, so the rate is about 1 in 17 runs here. The test code is
the same on 85dc7239d (the lane 7 test commit), so the fix did not cause it. It matters now because the
test stops being an expected failure and becomes a normal test in make test-integration (which uses
-race) on the branch that will be merged.
Suggested fix: remove h.DrainMail() from the caller and keep the one in the check (:76). That check
already waits for every mail of the round before it counts.
NOTE 1: NOW() inside the transaction is the transaction start, before the lock wait. A waiter's
count window and its created_at/expires_at are therefore early by the wait (milliseconds). This has
no practical effect.
NOTE 2 (outside the fix): Handler.Drain at shutdown has the same Add-during-Wait pattern against
requests that are still arriving. It is pre-existing and only shows in -race builds. I did not
investigate it further.
Gates
gofmt clean; go vet with and without -tags=integration on internal/auth/..., internal/server,
cmd/server; make complexity-check 0; make staticcheck 0; internal/repo tests ok; English check ok.
account_token.go has 172 lines. Commit type fix:, no !, no attribution lines. No allowlist entry, no
raised limit, no skipped test.
H-L7-02: Owner seed under an advisory lock
What the fix does
EnsureAdminUser keeps its unlocked checkExistingAdmin as the fast path. Only when no Owner exists
does createAdminAccount (admin_seed.go:102) hash the password, then run one transaction:
pg_advisory_xact_lock(hashtext('mnema.owner_seed')), re-check hasAdminUser, check the name, insert.
A server that finds an Owner inside the lock logs "an administrator was created by another server
starting at the same time" and returns nil. The name check moved inside the lock, so "taken by the
regular user X" is true again. No migration and no unique index; the commit message explains why (an
index would fail to build on a database that already has two Owners from this race).
Cause removed? Probes
Recorded test (both subtests) passes 5 of 5 runs; reverted, it fails 10 of 10 rounds in each subtest ("taken by a regular user" and "2 owners"). Own probes, on the fix and on the base:
| Probe | Fix | Base |
|---|---|---|
| Upgrade path: an Owner exists while another session holds the seed lock | returns nil in 3 ms (no lock taken) | 3 ms |
| Two development starts with an empty password, 10 rounds: exactly one generated password logged, by the server that created the Owner, and it signs in; one "another server" line; both starts nil | 0 of 10 bad | 10 of 10 bad (loser fails with the false message) |
Names that differ only in case (Owner, owner) | 0 of 10 bad, one Owner | 10 of 10 bad |
| Name of an existing regular user, no Owner | ADMIN_USERNAME "Member" is taken by the regular user "member" (true) | same |
| The seeding server dies inside the transaction (its backend terminated after it took the lock and inserted an uncommitted Owner) | the waiter gets the lock, seeds its own Owner, returns nil after 0.44 s; one Owner (the waiter's) | (no lock) |
| The seeding server hangs inside the transaction (2 s statement timeout) | the waiter fails with lock admin seed: … statement timeout (57014) after 2.0 s, no Owner | (no lock) |
The generated password is logged only after created and the commit, so a server that loses never logs
one (admin_seed.go:122-127; probe above). A server that dies releases the lock and rolls back its insert
when its connection closes.
Lock key
hashtext('mnema.owner_seed') = 1629348251. It differs from the other static hashtext keys
(mnema.event_role_templates -51764638, wiki_images 294820243, mnema.wiki -375449778). It cannot
equal any constant key, because those are all above 2^31. It can collide only with a per-user
'staged:' || user key, with probability 2^-32 per user, and only at the first start. At that point no
user exists, so no upload staging can hold such a key. ADMIN_USERNAME is read nowhere else at run time
(grep), so a loser with a different name has no other effect.
Findings
NOTE 3: When the seeding server hangs (does not die), a waiter fails its start after
DB_STATEMENT_TIMEOUT (15 s), and the holder is cut off only by the 1-minute
idle_in_transaction_session_timeout. That can mean a few restarts in a row. This is only an artificial
hang (the real holder runs three short statements) and the restart policy covers it.
NOTE 4 (unchanged behaviour): A losing server whose seed settings are invalid (for example an empty
ADMIN_INITIAL_PASSWORD in production) still fails before the lock with that error, where a later start
would only warn. This is the same as before the fix, and the docs sentence ("start normally, as any
later start does") is true for valid settings.
Docs: the doc/roles-and-permissions.md sentence matches. Gates as for 01, all green; admin_seed.go has 157 lines.
H-L7-03: role templates counted and created under a lock
What the fix does
CreateTemplate (store_templates.go:75) runs the count and the insert in inTx after
pg_advisory_xact_lock(hashtext('mnema.event_role_templates')). The unique index on the name still maps
to the same 409 "a template called X exists". The cap error and its 409 are unchanged. No other code
path inserts templates (only migration 0086 seeds the presets). Update and delete cannot raise the count.
The production Store.DB is the pool (router_build_services.go:114), which inTx needs. Other paths of
the store already depend on that.
Probes
Recorded test passes 5 of 5 runs; reverted, it fails 10 of 10 rounds (52 to 57 templates, 3 to 8 creates). Own probes, on the fix and on the base:
| Probe | Fix | Base |
|---|---|---|
| 8 creates with the same name at 30 templates: 1 created, 7 x 409 "a template called Same name exists" | 0 of 10 bad | 0 of 10 bad (unchanged) |
| At 50: 3 deletes and 8 creates at once; final count <= 50 and equal to 47 + creates | 0 of 10 bad | 0 of 10 bad (not discriminating: the deletes make room; shows no regression) |
Through the handler (createTemplate as the Owner), 8 at once at 49: one 201, seven 409 | 0 of 10 bad | 10 of 10 bad (8 created, 57 templates) |
Lock key: hashtext('mnema.event_role_templates') = -51764638. It differs from every other static key
and cannot equal a constant. A collision with a per-user staged: key (probability 2^-32 per user)
would only make two short transactions wait for each other. Neither transaction takes a second advisory
lock, so no wait cycle is possible.
No findings. Gates as for 01, all green; store_templates.go has 160 lines.
What I ran
All with DOCKER_HOST set to the local OrbStack socket, nice -n 10, -p 1.
- Per branch: gofmt;
go vetandgo vet -tags=integration(internal/auth/..., internal/communityevents, internal/server, cmd/server);make complexity-check;make staticcheck;go test ./internal/repo/;node scripts/check-english-only.mjs. All exit 0. - Recorded race tests 5 times per fix (01: both store tests and the route test; 02; 03). All pass.
- Revert check per fix: production files reset to 85dc7239d, tests kept. 01 needed a one-line
ErrTooManyLinksstub so the tests build. Every recorded test fails in every round. - Full integration suites with
-race -count=1:- 01: internal/auth/... and cmd/server: all ok except the race-detector failure in SHOULD-FIX 1
(reruns clean: 12 isolated runs and 4 full
authnruns). internal/server: ok (489 s). - 02: internal/auth/..., cmd/server, internal/server: all ok, no data race.
- 03: internal/communityevents (unit and integration), cmd/server, internal/server: all ok, no data race.
- No "can't assign requested address" failures occurred.
- 01: internal/auth/... and cmd/server: all ok except the race-detector failure in SHOULD-FIX 1
(reruns clean: 12 isolated runs and 4 full
- Unit tests
go test -race ./internal/auth/...on 01 and 02: ok. - The probes in the tables above, on each fix and on 85dc7239d.
What I did not check
- More than one real server process: H-L7-02 was probed with two seeders in one process and a terminated backend, not with two binaries.
- The SFU, web and e2e suites (the fixes touch none of them).
make openapi-checkwas not run (no annotation changed). - Production
Handler.Drainbeyond NOTE 2. - I did not mutate the fixes beyond the revert.