Independent review of pull request 798 (export download)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: an agent that did not write the change. Read-only: nothing in the change was edited; the probes and the temporary reverts ran in a throwaway git worktree (and one of origin/main) that were removed afterwards, against a local throwaway PostgreSQL in Docker. No real instance was contacted.

Reviewed: one commit on top of origin/main (e6c9d9394), head 48819231d ("a data export stays downloadable for an hour after its download and is sent with its length"): internal/auth/account/account_export.go, the new account_export_download.go, internal/auth/wire_types.go, the tests, the OpenAPI document, SECURITY.md, doc/operations.md, the web strings and types. Read around it: the build (account_export_build.go), the sweep, the erasure steps (account_erase_steps.go, account_delete.go), the orphan rule (internal/media/orphans.go), AccountData.vue and its test, the desktop proxy's answer limits (mnema-talk-desktop, experimental/app/src/web_proxy/forward.rs:250-277, limits.rs).

Verdict: MERGE AFTER FIXES

No BLOCKER. The change does what it says, and it does fix the reported problem. With a Content-Length the desktop proxy reads up to response_cap (at least 128 MiB, above the 64 MiB export bound) and no longer stops at the 32 MiB it allows a body of unknown length. A copy the host loses after the last byte can now be downloaded again. The state machine, the erasure, the daily limit and the cut-transfer paths hold under the probes below.

One SHOULD-FIX is older than this change: the single-download claim does not hold under parallel requests. The change rewrites claimDownload, keeps the promise "only one download holds it at a time" in SECURITY.md, and lists it as unchanged, so it should be fixed here. The fix is three lines and I verified it. Two smaller SHOULD-FIX items come with the new hour and the new Content-Length.

BLOCKER

None.

SHOULD-FIX

S1. Parallel downloads all win the "single-download" claim (exists on main too)

internal/auth/account/account_export_download.go:97-104 (claimDownload).

  • The claim is UPDATE … WHERE id = (SELECT id … AND (claimed_at IS NULL OR claimed_at < NOW() - 10 min) … LIMIT 1). Under READ COMMITTED the subquery is evaluated once, with the statement's snapshot. A second caller that waited on the row lock re-checks only id = <that id> against the new row version, not the claim condition, so it also succeeds. This is the codebase's own lesson: no condition in a subquery under a row lock.

  • Evidence (scratch integration test, six goroutines released together, claimed_at reset between rounds): on the branch, 29 of 30 rounds let all six callers claim (round 1: 6 callers claimed the export at once … 29 of 30 rounds let more than one download hold the claim). The same probe against origin/main (its three-value claimDownload): 30 of 30 rounds.

  • Consequence: SECURITY.md ("only one download holds it at a time") and the PR description ("the 10-minute single-download claim" unchanged) are not true. With the hour, extra copies within the hour are allowed anyway, so the harm is small: parallel transfers of one archive, each audited. Even so, the documented invariant of a protected area is false.

  • Smallest fix I accept (verified: the probe then passes in all 30 rounds, and the existing and new export tests stay green): keep the subquery only to pick the newest ready row, and put every condition in the outer WHERE, which PostgreSQL re-checks on the locked row:

    UPDATE data_exports SET claimed_at = NOW()
    WHERE id = (SELECT id FROM data_exports WHERE user_id = $1 AND status = 'ready'
                ORDER BY requested_at DESC LIMIT 1)
      AND status = 'ready' AND s3_key IS NOT NULL AND expires_at > NOW()
      AND (downloaded_at IS NULL OR downloaded_at > NOW() - $2 * INTERVAL '1 second')
      AND (claimed_at IS NULL OR claimed_at < NOW() - INTERVAL '10 minutes')
    RETURNING id, s3_key, size_bytes
    

    The repo's other pattern is just as good: lock the newest row FOR UPDATE in a transaction, then claim it in a new statement. Add the parallel test (racetest style) so this stays fixed.

S2. A response shorter than its Content-Length counts as a complete download

internal/auth/account/account_export_download.go:70-79.

  • Content-Length comes from size_bytes and the body comes from the store. size_bytes is counted.n, the bytes the archive writer passed into the upload pipe (account_export_build.go:111,133-135), so the two agree as long as the store returns what it took. I found no code path that makes them differ. They differ only if the stored object is truncated or replaced.
  • What each side then sees (scratch test over a real httptest.Server):
    • Object 100 bytes longer than size_bytes: the first Write exceeds the declared length and Go writes nothing (http.ErrContentLength). The handler releases the claim and the client gets unexpected EOF. The export is not counted. This is safe.
    • Object 100 bytes shorter: io.Copy returns no error, so the server records downloaded_at, writes auth.export_download and starts the hour. The client reads 5356 of 5456 bytes and gets unexpected EOF. Every retry in the hour fails the same way. Before this change there was no Content-Length, so a client always got the whole object.
  • Smallest fix: compare the count. n, err := io.Copy(w, body); if err == nil && n != size { err = io.ErrUnexpectedEOF } treats the transfer as interrupted. The current branch logs that, releases the claim and records nothing.
  • Range, compression, HTTP/2: no compression middleware sits on the API routes (none in internal/server, internal/httpx), so nothing rewrites the length. Range is ignored: the answer is always the whole file with 200, which is correct with Content-Length. On HTTP/2, Go enforces the declared length the same way as on HTTP/1.1.

S3. The sweep removes the file under a running download, and the archive's own README is now wrong

  • internal/auth/account/account_export.go:272-273: the sweep's new branch (downloaded_at < NOW() - 1 h) ignores claimed_at. Probe: complete download; downloaded_at moved to 59 minutes ago; a second download claims; the hour ends; the sweep runs. Result {downloaded:true claimed:true hasKey:false} files=0. With the S3 client an already-open GET usually keeps streaming after a DELETE, but that depends on the store, and I did not check it on SeaweedFS or MinIO. The same edge existed at the 24-hour expiry. The hour makes it common: any second download started in the hour's last minutes of a slow link. Smallest fix: `AND (claimed_at IS NULL OR claimed_at < NOW()
    • INTERVAL '10 minutes')` on the sweep's selection. A claim is at most ten minutes, so the file goes at the next sweep after the claim ends.
  • internal/auth/account/account_export_build.go:98: the README inside every archive still says "This file was available for download once, for 24 hours." That is the member's own record of the retention, and it is now false. Change it to the new rule (for example "This file could be downloaded for 24 hours; it was removed one hour after the first complete download.").

NOTE

  • Audit outside the transaction (unchanged from main). finishDownload (account_export_download.go:116-119) and AuditUser (account_export_download.go:79) are two separate statements, and errors are ignored. Every complete download is audited, including repeats in the hour, but not atomically with downloaded_at. The codebase's rule is audit inside the transaction: one pgx.BeginFunc with the UPDATE and h.Audit.RecordTx (as eraseInTransaction does) closes it. Not required for this change.
  • finishDownload clears any claim. claimed_at = NULL is unconditional, so a download that ran past its ten-minute claim clears a newer claim. After S1 it matters little; AND claimed_at = <own claim time> would make it exact.
  • Stolen session. SECURITY.md says the hour "gives a stolen session no file it could not already take in the 24 hours before the member's own download". That holds for a session stolen before the download. A session stolen after it gains one hour it did not have before. The archive's content is what the same session can mostly read through the API anyway, and every download is audited, so I accept the trade-off. Suggested wording: "The hour gives a stolen session at most one more hour to take the same file; …".
  • Retention wording. doc/operations.md:1684-1685 says "one hour after the first complete download, at the latest". The sweep runs every 15 minutes and waits for an online backup, so the file can stay up to about 1 h 15 min. The "at most 24 hours" before it has the same slack. "Removed by the next sweep (every 15 minutes) after that hour" is exact.
  • German intro text. In web/src/i18n/de.json exportIntro, the pronoun for the download has no matching noun earlier in the paragraph (the paragraph opens with the feminine word for a ZIP file). Name the export instead of using the pronoun.
  • The web UI does not use downloaded_at. After a download the button again reads "Download (valid until HH:MM)" with the end of the hour. That is correct, but it does not say the file was downloaded already. Optional.
  • Test hygiene. TestATransferCutInTheMiddleKeepsTheExport (account_export_download_integration_test.go:126-147): when the status or length check fails, t.Fatalf runs before resp.Body.Close(). The deferred srv.Close() then waits for a handler blocked writing 32 MiB to a reader that never reads, and the package hangs until the 10-minute test timeout (seen with the behaviour reverted). Add defer resp.Body.Close() right after http.Get.
  • Swag order. In account_export_download.go the @Header 200 Content-Length line sits between the @Failure lines. That is harmless (the generated document is right) but easy to misread.
  • Older edge, unchanged. expires_at is ready_at + 24 h and the daily limit runs from requested_at. For the length of one build (at most five minutes) a new export can be requested while the old one is still downloadable. The hour does not widen this: a downloaded export's window ends at the earlier of the two.

Checked and found right

  • State names (account_export.go:186-198): ready while a key exists and the earlier of expiry and downloaded_at + 1 h is in the future; then downloaded (downloaded) or expired. A row downloaded before the upgrade (key already NULL) reads downloaded. ExpiresAt is capped by the hour (minTime), so old web bundles show the right end time.
  • A second complete download does not move the hour (COALESCE). A cut transfer after the first complete one releases the claim and keeps the key and the first downloaded_at (probe passed).
  • The daily limit still holds during and after the hour: claimExport counts every non-failed row from the last 24 hours, whatever its download state. The hour cannot produce two different exports.
  • Erasure during the hour: the erasure step DELETE … WHERE s3_key IS NOT NULL RETURNING s3_key takes the kept file, and the row is gone (probe: rows=0, files=0, the download then answers 404). During a download, the in-flight finishDownload matches no row, and the audit insert for the erased account is refused by the guard, as on main. The orphan cleanup leaves the kept file alone because its row still names it.
  • Prefetch: the check runs before any lookup, so a 403 is the same for every member and reveals nothing. A user's click on an <a download> never carries Sec-Purpose/Purpose/X-Moz "prefetch", so a real download is not mislabelled. Sec-Fetch-Site, Cache-Control: no-store, Content-Disposition, nosniff and GET-only (HEAD 405) are unchanged.
  • Old clients: an old web bundle sees ready again after a download and shows the download button with the end of the hour. It polls only while pending and downloads only on a click, so there is no loop.
  • API: downloaded_at is optional and additive. The state enum is unchanged. The OpenAPI document is regenerated. rest.ts validates the new field.
  • Do the new tests fail without the change? With the production behaviour reverted in place (old claim, file dropped on finish, no Content-Length, no prefetch check, old state order and sweep), these fail: TestACompleteDownloadCarriesItsLengthAndCanBeRepeatedForAnHour (no Content-Length), TestTheHourNeverOutlastsTheExpiry, TestATransferCutInTheMiddleKeepsTheExport (length -1, then the hang above), TestAPrefetchDoesNotTouchTheExport (200), and in internal/server TestAnExportHoldsOnlyTheMembersOwnData (second download 404). TestTheFileIsGoneAnHourAfterTheDownload, TestAHeadRequestTakesNothing and TestADownloadDoesNotChangeTheDailyLimit pass either way. They guard rules that did not change.

Gates run (exit codes read before any output)

All of these exited 0 on the branch head:

  • go build ./..., go vet ./..., go vet -tags=integration ./...
  • make staticcheck, make complexity-check
  • make openapi-check, make openapi-diff ("no breaking changes since v3.0.10"), make contracts-check
  • make docs-check, make english-check
  • go test ./internal/repo/
  • go test -p 1 -tags=integration ./internal/auth/... (auth, account, authn, mailtext, memberadmin)
  • go test -p 1 -tags=integration -run Export ./internal/server/
  • npm run lint (including the strict i18n check), npm run typecheck
  • vitest for src/settings/components/AccountData.test.ts and src/shared/lib (76 files, 1689 tests passed)

The review probes (parallel claims, sweep under a claim, length mismatch both ways, erasure in the hour, cut transfer after the first download) ran only in the scratch worktree. S1 and S3 failed there as described. The others passed.

Not checked

  • Whether SeaweedFS or MinIO cut an open GET when the object is deleted (S3).
  • The desktop branch fix/export-download and the webview's save path. I read only the proxy's answer limits to confirm that the Content-Length fix reaches it.
  • A real browser's prefetch or speculation-rules behaviour against the route. The header rules are read from the code, not observed.
  • The nightly deep race sweep (RACE_SWEEP=1). The parallel probe ran 30 rounds.

Round 2: head 8c0244c90 (on 5b6dcd47f)

Same rules as round 1: read-only, a fresh throwaway worktree (removed afterwards), local Docker PostgreSQL only. Reviewed: 5b6dcd47f (the download attribute names the file) and 8c0244c90 (the fixes for S1 to S3 and the notes).

Verdict: MERGE

Every round-1 SHOULD-FIX is fixed. Each fix except the audit transaction has a test that fails when that fix alone is reverted. The claim token holds up under the checks below. What is left are notes; none blocks the merge.

Each finding, its fix and its test

For each fix I reverted only that fix in the scratch worktree and ran its new test; the test failed. With the fix restored, it passed.

Round-1 findingFix in 8c0244c90Test, and what it says with the fix reverted
S1 parallel claimsclaimDownload (account_export_download.go:114-126): the subquery only picks the newest ready row; status, key, expiry, the hour and the claim lapse are in the outer WHERETestOnlyOneOfParallelDownloadsHoldsTheExport (racetest, 6 callers): "10 of 10 rounds broke the invariant: round 0: 2 downloads …, round 1: 6 downloads …"
S2 short bodyn != c.size is treated as a cut transfer (:73-83)TestABodyShorterThanItsLengthDoesNotSpendTheExport: {downloaded:true claimed:false hasKey:true}
S3 sweep under a claimthe sweep skips a claim younger than 10 minutes (account_export.go:272-274)TestTheSweepLeavesAFileWhileADownloadHoldsIt: {downloaded:true claimed:true hasKey:false} files=0
S3 archive READMEaccount_export_build.go:98-99TestTheArchiveSaysHowLongItCanBeDownloaded: fails on the old sentence
Note: finish clears any claimclaim token: claimed_at through RETURNING; release and finish match it (:128-154)TestAnOldDownloadFinishingLeavesANewerOnesClaim: {downloaded:true claimed:false …}
Note: audit outside the transactionfinishDownload writes the UPDATE and Audit.RecordTx in one pgx.BeginFunc; no entry when the row is gone (erasure)TestACompleteDownloadIsAuditedWithItsRecord still passes when the audit is moved back to AuditUser after the transaction: it counts entries, not atomicity. See N1.
Notes: SECURITY.md, operations.md, German intro, test hangwording corrected; defer resp.Body.Close() addedchecked by reading. The texts are now true; "den Export" is correct German

5b6dcd47f changes only the link: download becomes download="mnema-talk-export.zip", plus a web test for the attribute. The request is still the same-origin GET with the same headers, so the Sec-Fetch-Site and prefetch rules see no difference. Content-Disposition names the same file. Nothing else changed in that commit.

The claim token

  • Precision. claimed_at is TIMESTAMPTZ (0043_account_privacy.sql:25), stored in microseconds. pgx moves it in the binary format as an int64 of microseconds in both directions, and no JSON is on the path: the token lives only inside the request. A probe ran 300 rounds of claim and release with the token from RETURNING, then a finish. Every release freed the claim, the finish recorded the download and freed the claim, and no token had a sub-microsecond part. The time zone of the scanned time.Time does not matter: the value compares as an instant.
  • clock_timestamp() against NOW(). The claim stores clock_timestamp(), which is a little later than the statement's NOW(), and the lapse compares with NOW() of later statements. Only the database clock is involved; the app clock is not. A clock stepped back makes a fresh claim last longer than ten minutes: the member gets 404 until the clock catches up, and nothing leaks. A clock stepped forward lets a claim lapse early, so a second download can claim while the first still runs. The tokens differ, so neither ends the other's hold, and both transfers are whole copies of the same file, each audited.
  • The 10-minute edge. The server's WriteTimeout is 10 minutes (internal/server/run.go:27). It starts before the handler claims, so a transfer is cut by the server before or at the moment its claim lapses. A second holder at the edge therefore overlaps a dying transfer by at most milliseconds. When the cut transfer releases, its token no longer matches and it leaves the new claim alone.
  • The sweep and a new claim at the same time. They cannot both act on one row. The sweep only selects rows past their hour or their expiry. A new claim needs a row inside the hour and before its expiry. The two sets do not overlap at any instant. The one remaining gap is the sweep's select, then its object delete, then s3_key = NULL, without re-checking the claim. A claim cannot start on such a row in between, because the row is already past its hour.

Notes (round 2)

  • N1. No test pins the audit to the transaction. Reverting to the old AuditUser after the commit keeps every test green. A test that makes the audit insert fail (for example a CHECK added to audit_log in the test) and expects downloaded_at to stay NULL would pin it. Optional.
  • N2. A failed audit insert leaves the claim held. The rollback also undoes claimed_at = NULL, so the member gets 404 for up to ten minutes after a complete copy, and nothing is recorded. The rollback is the right outcome for "audit in the transaction". A releaseDownload(c) after a failed transaction would end the wait early. Optional.

Gates and CI (round 2)

All of these exited 0 at 8c0244c90:

  • go build ./..., go vet ./..., go vet -tags=integration ./...
  • make staticcheck, make complexity-check
  • make openapi-check, make openapi-diff ("no breaking changes since v3.0.11"), make contracts-check
  • make docs-check, make english-check
  • go test ./internal/repo/
  • go test -p 1 -tags=integration ./internal/auth/...
  • go test -p 1 -tags=integration -run Export ./internal/server/
  • npm run lint, npm run typecheck
  • vitest for AccountData.test.ts and src/shared/lib (1689 tests passed)

While I was reviewing, the branch moved to 680f8e3c8 ("after a download the export card says when it was downloaded and until when it can be downloaded again"). That commit is web-only: an exportDownloadedAt sentence in English and German, shown while downloaded_at is set, and its test. It answers the optional UI note of round 1. I read it: the v-if chain stays valid and both texts are true. npm run lint, npm run typecheck, make english-check and the AccountData.test.ts vitest (9 tests) exited 0 on it.

CI (gh pr checks 798) on head 680f8e3c8: all 41 checks passed, merge state CLEAN. That covers the backend checks and integration suites (realtime, rest, server), coverage, CodeQL, govulncheck, the docker image, the e2e runs (Firefox, WebKit and the four shards), the WebKit macOS call runs, both proxies, the setup assistant and all seven upgrade drills.

Not checked (round 2)

  • A real clock step on the database host. The analysis above comes from reading the SQL.
  • The deep race sweep (RACE_SWEEP=1) myself. The smoke run (10 rounds) passed, and with the S1 fix reverted it failed in 10 of 10 rounds.

Round 2 addendum: head 680f8e3c8

Heads reviewed in round 2:

  • 8c0244c90 for the server fixes.
  • 680f8e3c8 for the web note on top of it. This is the PR head when this report was finished.

680f8e3c8 changes only web/ (AccountData.vue, its test, en.json, de.json).

  • Texts.

    • English exportDownloadedAt: "Downloaded on {at}. You can download it again until {until}; then it is removed."
    • German (de.json, same key): the same two statements, naming the export explicitly.

    Both are true for the server's behaviour. {until} is the capped expires_at: the end of the hour, or the 24-hour expiry if that comes first. Both use the same when() date and time format as the button. The German is grammatical and uses "den Export", matching the corrected intro.

  • Clipping. The note is a plain <p class="text-sm"> in the card's flex column (.settings-card: flex-direction: column, no fixed height, no overflow). Neither it nor .btn (min-height: 2rem, no nowrap, overflow or ellipsis) can truncate, so long German dates wrap. I checked this in the markup and CSS. I did not render it in a browser.

  • Old server without downloaded_at. The note sits behind v-if="status.downloaded_at", so it does not appear. An older server answers downloaded straight after a download, and the card shows the existing "downloaded and removed" text. The vitest checks that the note is absent while downloaded_at is missing.

  • Gates on 680f8e3c8.

    • The AccountData.test.ts vitest passed (9 tests). It covers the note's text pattern, that no { placeholder is left, and the button still saying "valid until".
    • npm run lint (strict i18n check), npm run typecheck and make english-check exited 0.
    • CI on 680f8e3c8: all 41 checks passed, merge state CLEAN.

The verdict stays MERGE.

Something wrong on this page? Tell us.