Independent review of pull request 798 (export download)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: an agent that did not write the change. Read-only: nothing in the
change was edited; the probes and the temporary reverts ran in a throwaway git
worktree (and one of origin/main) that were removed afterwards, against a
local throwaway PostgreSQL in Docker. No real instance was contacted.
Reviewed: one commit on top of origin/main (e6c9d9394), head 48819231d
("a data export stays downloadable for an hour after its download and is sent
with its length"): internal/auth/account/account_export.go, the new
account_export_download.go, internal/auth/wire_types.go, the tests, the
OpenAPI document, SECURITY.md, doc/operations.md, the web strings and types.
Read around it: the build (account_export_build.go), the sweep, the erasure
steps (account_erase_steps.go, account_delete.go), the orphan rule
(internal/media/orphans.go), AccountData.vue and its test, the desktop
proxy's answer limits (mnema-talk-desktop,
experimental/app/src/web_proxy/forward.rs:250-277, limits.rs).
Verdict: MERGE AFTER FIXES
No BLOCKER. The change does what it says, and it does fix the reported
problem. With a Content-Length the desktop proxy reads up to response_cap
(at least 128 MiB, above the 64 MiB export bound) and no longer stops at the
32 MiB it allows a body of unknown length. A copy the host loses after the last
byte can now be downloaded again. The state machine, the erasure, the daily
limit and the cut-transfer paths hold under the probes below.
One SHOULD-FIX is older than this change: the single-download claim does not
hold under parallel requests. The change rewrites claimDownload, keeps the
promise "only one download holds it at a time" in SECURITY.md, and lists it as
unchanged, so it should be fixed here. The fix is three lines and I verified
it. Two smaller SHOULD-FIX items come with the new hour and the new
Content-Length.
BLOCKER
None.
SHOULD-FIX
S1. Parallel downloads all win the "single-download" claim (exists on main too)
internal/auth/account/account_export_download.go:97-104 (claimDownload).
The claim is
UPDATE … WHERE id = (SELECT id … AND (claimed_at IS NULL OR claimed_at < NOW() - 10 min) … LIMIT 1). Under READ COMMITTED the subquery is evaluated once, with the statement's snapshot. A second caller that waited on the row lock re-checks onlyid = <that id>against the new row version, not the claim condition, so it also succeeds. This is the codebase's own lesson: no condition in a subquery under a row lock.Evidence (scratch integration test, six goroutines released together,
claimed_atreset between rounds): on the branch, 29 of 30 rounds let all six callers claim (round 1: 6 callers claimed the export at once…29 of 30 rounds let more than one download hold the claim). The same probe againstorigin/main(its three-valueclaimDownload): 30 of 30 rounds.Consequence: SECURITY.md ("only one download holds it at a time") and the PR description ("the 10-minute single-download claim" unchanged) are not true. With the hour, extra copies within the hour are allowed anyway, so the harm is small: parallel transfers of one archive, each audited. Even so, the documented invariant of a protected area is false.
Smallest fix I accept (verified: the probe then passes in all 30 rounds, and the existing and new export tests stay green): keep the subquery only to pick the newest ready row, and put every condition in the outer
WHERE, which PostgreSQL re-checks on the locked row:UPDATE data_exports SET claimed_at = NOW() WHERE id = (SELECT id FROM data_exports WHERE user_id = $1 AND status = 'ready' ORDER BY requested_at DESC LIMIT 1) AND status = 'ready' AND s3_key IS NOT NULL AND expires_at > NOW() AND (downloaded_at IS NULL OR downloaded_at > NOW() - $2 * INTERVAL '1 second') AND (claimed_at IS NULL OR claimed_at < NOW() - INTERVAL '10 minutes') RETURNING id, s3_key, size_bytesThe repo's other pattern is just as good: lock the newest row
FOR UPDATEin a transaction, then claim it in a new statement. Add the parallel test (racetest style) so this stays fixed.
S2. A response shorter than its Content-Length counts as a complete download
internal/auth/account/account_export_download.go:70-79.
- Content-Length comes from
size_bytesand the body comes from the store.size_bytesiscounted.n, the bytes the archive writer passed into the upload pipe (account_export_build.go:111,133-135), so the two agree as long as the store returns what it took. I found no code path that makes them differ. They differ only if the stored object is truncated or replaced. - What each side then sees (scratch test over a real
httptest.Server):- Object 100 bytes longer than
size_bytes: the firstWriteexceeds the declared length and Go writes nothing (http.ErrContentLength). The handler releases the claim and the client getsunexpected EOF. The export is not counted. This is safe. - Object 100 bytes shorter:
io.Copyreturns no error, so the server recordsdownloaded_at, writesauth.export_downloadand starts the hour. The client reads 5356 of 5456 bytes and getsunexpected EOF. Every retry in the hour fails the same way. Before this change there was no Content-Length, so a client always got the whole object.
- Object 100 bytes longer than
- Smallest fix: compare the count.
n, err := io.Copy(w, body); if err == nil && n != size { err = io.ErrUnexpectedEOF }treats the transfer as interrupted. The current branch logs that, releases the claim and records nothing. - Range, compression, HTTP/2: no compression middleware sits on the API routes
(none in
internal/server,internal/httpx), so nothing rewrites the length.Rangeis ignored: the answer is always the whole file with 200, which is correct with Content-Length. On HTTP/2, Go enforces the declared length the same way as on HTTP/1.1.
S3. The sweep removes the file under a running download, and the archive's own README is now wrong
internal/auth/account/account_export.go:272-273: the sweep's new branch (downloaded_at < NOW() - 1 h) ignoresclaimed_at. Probe: complete download;downloaded_atmoved to 59 minutes ago; a second download claims; the hour ends; the sweep runs. Result{downloaded:true claimed:true hasKey:false} files=0. With the S3 client an already-open GET usually keeps streaming after a DELETE, but that depends on the store, and I did not check it on SeaweedFS or MinIO. The same edge existed at the 24-hour expiry. The hour makes it common: any second download started in the hour's last minutes of a slow link. Smallest fix: `AND (claimed_at IS NULL OR claimed_at < NOW()- INTERVAL '10 minutes')` on the sweep's selection. A claim is at most ten minutes, so the file goes at the next sweep after the claim ends.
internal/auth/account/account_export_build.go:98: the README inside every archive still says "This file was available for download once, for 24 hours." That is the member's own record of the retention, and it is now false. Change it to the new rule (for example "This file could be downloaded for 24 hours; it was removed one hour after the first complete download.").
NOTE
- Audit outside the transaction (unchanged from main).
finishDownload(account_export_download.go:116-119) andAuditUser(account_export_download.go:79) are two separate statements, and errors are ignored. Every complete download is audited, including repeats in the hour, but not atomically withdownloaded_at. The codebase's rule is audit inside the transaction: onepgx.BeginFuncwith theUPDATEandh.Audit.RecordTx(aseraseInTransactiondoes) closes it. Not required for this change. finishDownloadclears any claim.claimed_at = NULLis unconditional, so a download that ran past its ten-minute claim clears a newer claim. After S1 it matters little;AND claimed_at = <own claim time>would make it exact.- Stolen session. SECURITY.md says the hour "gives a stolen session no file it could not already take in the 24 hours before the member's own download". That holds for a session stolen before the download. A session stolen after it gains one hour it did not have before. The archive's content is what the same session can mostly read through the API anyway, and every download is audited, so I accept the trade-off. Suggested wording: "The hour gives a stolen session at most one more hour to take the same file; …".
- Retention wording.
doc/operations.md:1684-1685says "one hour after the first complete download, at the latest". The sweep runs every 15 minutes and waits for an online backup, so the file can stay up to about 1 h 15 min. The "at most 24 hours" before it has the same slack. "Removed by the next sweep (every 15 minutes) after that hour" is exact. - German intro text. In
web/src/i18n/de.jsonexportIntro, the pronoun for the download has no matching noun earlier in the paragraph (the paragraph opens with the feminine word for a ZIP file). Name the export instead of using the pronoun. - The web UI does not use
downloaded_at. After a download the button again reads "Download (valid until HH:MM)" with the end of the hour. That is correct, but it does not say the file was downloaded already. Optional. - Test hygiene.
TestATransferCutInTheMiddleKeepsTheExport(account_export_download_integration_test.go:126-147): when the status or length check fails,t.Fatalfruns beforeresp.Body.Close(). The deferredsrv.Close()then waits for a handler blocked writing 32 MiB to a reader that never reads, and the package hangs until the 10-minute test timeout (seen with the behaviour reverted). Adddefer resp.Body.Close()right afterhttp.Get. - Swag order. In
account_export_download.gothe@Header 200 Content-Lengthline sits between the@Failurelines. That is harmless (the generated document is right) but easy to misread. - Older edge, unchanged.
expires_atisready_at + 24 hand the daily limit runs fromrequested_at. For the length of one build (at most five minutes) a new export can be requested while the old one is still downloadable. The hour does not widen this: a downloaded export's window ends at the earlier of the two.
Checked and found right
- State names (
account_export.go:186-198):readywhile a key exists and the earlier of expiry anddownloaded_at + 1 his in the future; thendownloaded(downloaded) orexpired. A row downloaded before the upgrade (key already NULL) readsdownloaded.ExpiresAtis capped by the hour (minTime), so old web bundles show the right end time. - A second complete download does not move the hour (
COALESCE). A cut transfer after the first complete one releases the claim and keeps the key and the firstdownloaded_at(probe passed). - The daily limit still holds during and after the hour:
claimExportcounts every non-failed row from the last 24 hours, whatever its download state. The hour cannot produce two different exports. - Erasure during the hour: the erasure step
DELETE … WHERE s3_key IS NOT NULL RETURNING s3_keytakes the kept file, and the row is gone (probe: rows=0, files=0, the download then answers 404). During a download, the in-flightfinishDownloadmatches no row, and the audit insert for the erased account is refused by the guard, as on main. The orphan cleanup leaves the kept file alone because its row still names it. - Prefetch: the check runs before any lookup, so a 403 is the same for every
member and reveals nothing. A user's click on an
<a download>never carriesSec-Purpose/Purpose/X-Moz"prefetch", so a real download is not mislabelled.Sec-Fetch-Site,Cache-Control: no-store,Content-Disposition,nosniffand GET-only (HEAD 405) are unchanged. - Old clients: an old web bundle sees
readyagain after a download and shows the download button with the end of the hour. It polls only whilependingand downloads only on a click, so there is no loop. - API:
downloaded_atis optional and additive. Thestateenum is unchanged. The OpenAPI document is regenerated.rest.tsvalidates the new field. - Do the new tests fail without the change? With the production behaviour
reverted in place (old claim, file dropped on finish, no Content-Length, no
prefetch check, old state order and sweep), these fail:
TestACompleteDownloadCarriesItsLengthAndCanBeRepeatedForAnHour(no Content-Length),TestTheHourNeverOutlastsTheExpiry,TestATransferCutInTheMiddleKeepsTheExport(length -1, then the hang above),TestAPrefetchDoesNotTouchTheExport(200), and ininternal/serverTestAnExportHoldsOnlyTheMembersOwnData(second download 404).TestTheFileIsGoneAnHourAfterTheDownload,TestAHeadRequestTakesNothingandTestADownloadDoesNotChangeTheDailyLimitpass either way. They guard rules that did not change.
Gates run (exit codes read before any output)
All of these exited 0 on the branch head:
go build ./...,go vet ./...,go vet -tags=integration ./...make staticcheck,make complexity-checkmake openapi-check,make openapi-diff("no breaking changes since v3.0.10"),make contracts-checkmake docs-check,make english-checkgo test ./internal/repo/go test -p 1 -tags=integration ./internal/auth/...(auth, account, authn, mailtext, memberadmin)go test -p 1 -tags=integration -run Export ./internal/server/npm run lint(including the strict i18n check),npm run typecheck- vitest for
src/settings/components/AccountData.test.tsandsrc/shared/lib(76 files, 1689 tests passed)
The review probes (parallel claims, sweep under a claim, length mismatch both ways, erasure in the hour, cut transfer after the first download) ran only in the scratch worktree. S1 and S3 failed there as described. The others passed.
Not checked
- Whether SeaweedFS or MinIO cut an open GET when the object is deleted (S3).
- The desktop branch
fix/export-downloadand the webview's save path. I read only the proxy's answer limits to confirm that the Content-Length fix reaches it. - A real browser's prefetch or speculation-rules behaviour against the route. The header rules are read from the code, not observed.
- The nightly deep race sweep (
RACE_SWEEP=1). The parallel probe ran 30 rounds.
Round 2: head 8c0244c90 (on 5b6dcd47f)
Same rules as round 1: read-only, a fresh throwaway worktree (removed
afterwards), local Docker PostgreSQL only. Reviewed: 5b6dcd47f (the
download attribute names the file) and 8c0244c90 (the fixes for S1 to S3
and the notes).
Verdict: MERGE
Every round-1 SHOULD-FIX is fixed. Each fix except the audit transaction has a test that fails when that fix alone is reverted. The claim token holds up under the checks below. What is left are notes; none blocks the merge.
Each finding, its fix and its test
For each fix I reverted only that fix in the scratch worktree and ran its new test; the test failed. With the fix restored, it passed.
| Round-1 finding | Fix in 8c0244c90 | Test, and what it says with the fix reverted |
|---|---|---|
| S1 parallel claims | claimDownload (account_export_download.go:114-126): the subquery only picks the newest ready row; status, key, expiry, the hour and the claim lapse are in the outer WHERE | TestOnlyOneOfParallelDownloadsHoldsTheExport (racetest, 6 callers): "10 of 10 rounds broke the invariant: round 0: 2 downloads …, round 1: 6 downloads …" |
| S2 short body | n != c.size is treated as a cut transfer (:73-83) | TestABodyShorterThanItsLengthDoesNotSpendTheExport: {downloaded:true claimed:false hasKey:true} |
| S3 sweep under a claim | the sweep skips a claim younger than 10 minutes (account_export.go:272-274) | TestTheSweepLeavesAFileWhileADownloadHoldsIt: {downloaded:true claimed:true hasKey:false} files=0 |
| S3 archive README | account_export_build.go:98-99 | TestTheArchiveSaysHowLongItCanBeDownloaded: fails on the old sentence |
| Note: finish clears any claim | claim token: claimed_at through RETURNING; release and finish match it (:128-154) | TestAnOldDownloadFinishingLeavesANewerOnesClaim: {downloaded:true claimed:false …} |
| Note: audit outside the transaction | finishDownload writes the UPDATE and Audit.RecordTx in one pgx.BeginFunc; no entry when the row is gone (erasure) | TestACompleteDownloadIsAuditedWithItsRecord still passes when the audit is moved back to AuditUser after the transaction: it counts entries, not atomicity. See N1. |
| Notes: SECURITY.md, operations.md, German intro, test hang | wording corrected; defer resp.Body.Close() added | checked by reading. The texts are now true; "den Export" is correct German |
5b6dcd47f changes only the link: download becomes
download="mnema-talk-export.zip", plus a web test for the attribute. The
request is still the same-origin GET with the same headers, so the
Sec-Fetch-Site and prefetch rules see no difference. Content-Disposition
names the same file. Nothing else changed in that commit.
The claim token
- Precision.
claimed_atisTIMESTAMPTZ(0043_account_privacy.sql:25), stored in microseconds. pgx moves it in the binary format as an int64 of microseconds in both directions, and no JSON is on the path: the token lives only inside the request. A probe ran 300 rounds of claim and release with the token fromRETURNING, then a finish. Every release freed the claim, the finish recorded the download and freed the claim, and no token had a sub-microsecond part. The time zone of the scannedtime.Timedoes not matter: the value compares as an instant. clock_timestamp()againstNOW(). The claim storesclock_timestamp(), which is a little later than the statement'sNOW(), and the lapse compares withNOW()of later statements. Only the database clock is involved; the app clock is not. A clock stepped back makes a fresh claim last longer than ten minutes: the member gets 404 until the clock catches up, and nothing leaks. A clock stepped forward lets a claim lapse early, so a second download can claim while the first still runs. The tokens differ, so neither ends the other's hold, and both transfers are whole copies of the same file, each audited.- The 10-minute edge. The server's
WriteTimeoutis 10 minutes (internal/server/run.go:27). It starts before the handler claims, so a transfer is cut by the server before or at the moment its claim lapses. A second holder at the edge therefore overlaps a dying transfer by at most milliseconds. When the cut transfer releases, its token no longer matches and it leaves the new claim alone. - The sweep and a new claim at the same time. They cannot both act on one
row. The sweep only selects rows past their hour or their expiry. A new claim
needs a row inside the hour and before its expiry. The two sets do not
overlap at any instant. The one remaining gap is the sweep's select, then its
object delete, then
s3_key = NULL, without re-checking the claim. A claim cannot start on such a row in between, because the row is already past its hour.
Notes (round 2)
- N1. No test pins the audit to the transaction. Reverting to the old
AuditUserafter the commit keeps every test green. A test that makes the audit insert fail (for example aCHECKadded toaudit_login the test) and expectsdownloaded_atto stay NULL would pin it. Optional. - N2. A failed audit insert leaves the claim held. The rollback also
undoes
claimed_at = NULL, so the member gets 404 for up to ten minutes after a complete copy, and nothing is recorded. The rollback is the right outcome for "audit in the transaction". AreleaseDownload(c)after a failed transaction would end the wait early. Optional.
Gates and CI (round 2)
All of these exited 0 at 8c0244c90:
go build ./...,go vet ./...,go vet -tags=integration ./...make staticcheck,make complexity-checkmake openapi-check,make openapi-diff("no breaking changes since v3.0.11"),make contracts-checkmake docs-check,make english-checkgo test ./internal/repo/go test -p 1 -tags=integration ./internal/auth/...go test -p 1 -tags=integration -run Export ./internal/server/npm run lint,npm run typecheck- vitest for
AccountData.test.tsandsrc/shared/lib(1689 tests passed)
While I was reviewing, the branch moved to 680f8e3c8 ("after a download the
export card says when it was downloaded and until when it can be downloaded
again"). That commit is web-only: an exportDownloadedAt sentence in English
and German, shown while downloaded_at is set, and its test. It answers the
optional UI note of round 1. I read it: the v-if chain stays valid and both
texts are true. npm run lint, npm run typecheck, make english-check and
the AccountData.test.ts vitest (9 tests) exited 0 on it.
CI (gh pr checks 798) on head 680f8e3c8: all 41 checks passed, merge
state CLEAN. That covers the backend checks and integration suites (realtime,
rest, server), coverage, CodeQL, govulncheck, the docker image, the e2e runs
(Firefox, WebKit and the four shards), the WebKit macOS call runs, both proxies,
the setup assistant and all seven upgrade drills.
Not checked (round 2)
- A real clock step on the database host. The analysis above comes from reading the SQL.
- The deep race sweep (
RACE_SWEEP=1) myself. The smoke run (10 rounds) passed, and with the S1 fix reverted it failed in 10 of 10 rounds.
Round 2 addendum: head 680f8e3c8
Heads reviewed in round 2:
8c0244c90for the server fixes.680f8e3c8for the web note on top of it. This is the PR head when this report was finished.
680f8e3c8 changes only web/ (AccountData.vue, its test, en.json,
de.json).
Texts.
- English
exportDownloadedAt: "Downloaded on {at}. You can download it again until {until}; then it is removed." - German (
de.json, same key): the same two statements, naming the export explicitly.
Both are true for the server's behaviour.
{until}is the cappedexpires_at: the end of the hour, or the 24-hour expiry if that comes first. Both use the samewhen()date and time format as the button. The German is grammatical and uses "den Export", matching the corrected intro.- English
Clipping. The note is a plain
<p class="text-sm">in the card's flex column (.settings-card:flex-direction: column, no fixed height, nooverflow). Neither it nor.btn(min-height: 2rem, nonowrap,overfloworellipsis) can truncate, so long German dates wrap. I checked this in the markup and CSS. I did not render it in a browser.Old server without
downloaded_at. The note sits behindv-if="status.downloaded_at", so it does not appear. An older server answersdownloadedstraight after a download, and the card shows the existing "downloaded and removed" text. The vitest checks that the note is absent whiledownloaded_atis missing.Gates on
680f8e3c8.- The
AccountData.test.tsvitest passed (9 tests). It covers the note's text pattern, that no{placeholder is left, and the button still saying "valid until". npm run lint(strict i18n check),npm run typecheckandmake english-checkexited 0.- CI on
680f8e3c8: all 41 checks passed, merge state CLEAN.
- The
The verdict stays MERGE.