Independent review of pull request 789 (follow-up for 3.0.8)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: an agent that did not write the change. Read-only: nothing in the change was edited; mutations and scratch tests ran in a temporary git worktree that was removed afterwards.

Reviewed: four commits on top of v3.0.7 (c7fafa242), head 435356e11: the review reports, storage health, the per-member timers, fail-closed permission enforcement.

Verdict: MERGE AFTER FIXES

No BLOCKER was found. I could not construct a step-by-step scenario in which the change loses a seat, leaks a lock, double-counts voice time or leaves a member with a right a change took away, short of a second panic inside the clean-up (which the change documents as ending the process). The four SHOULD-FIX items are small, and two of them (S1, S4) are untested or silently-disabled code paths that are cheap to close before merge.

BLOCKER

None.

SHOULD-FIX

S1. The grace timer's "end the presence again" branch is untested and its comment is inaccurate

internal/ws/voice_presence.go:121-136 (claimed, onPanic).

  • The comment says only a panic before the seat was removed makes onPanic end the presence again. The code cannot tell the two cases apart: once claimed is set, onPanic runs end() after any panic, including a panic in a listener after the seat is gone. That second endPresence is then a no-op (wasIn false), so it is harmless, but the comment describes something the code does not check.
  • Evidence that the branch is untested: in a scratch copy I replaced onPanic by safego.Abandon and deleted the retry. TestPanicInOneMembersGraceTimerLeavesTheHubConsistent, TestGraceTimerPanicStillTellsTheOthers and the other new tests still passed. Both tests panic in h.Presence, which runs after the seat was removed, so the retry never matters in them.
  • What a panic before removal could be is thin: inside the locked section only seatedLocked and map operations run. If one of them did panic deterministically, the retry panics the same way and, as documented, ends the process, so the branch adds little. Either test it with an injected panic in the keep function (a nil entry in h.rooms[ch] makes seatedLocked panic), or remove the branch and say in the comment that a panic after the removal is abandoned and one before it is not expected.
  • Smaller consequence of the same code: a retry runs endPresence a second time. If the member left and a new grace for the same key started in the few microseconds between the two calls, the second call would end that newer seat early (keep only looks at connections, not at a newer grace). The window is negligible; it is mentioned so the retry is not mistaken for free.

S2. The fail-closed clean-up is not isolated per member, and it does not remove a seat that has no connection and no grace

internal/ws/voice_kick.go:191-217 (disconnectForRecheck).

  • The seat loop is guarded per member (ws_perm_enforce_seat, Abandon), the disconnect loop is not: for id := range users { h.DisconnectUser(id) } runs unguarded. A panic in one member's DisconnectUser (c.close() -> conn.Close()) skips the disconnects of every member after them and the whole seat loop. On the HTTP path safego.handle re-panics (panic(again)) into net/http's recovery (internal/httpx/middleware.go:61), so the process continues with some members not re-checked. On timers the process exits. Same guard as for the seats would make the clean-up fail closed per member.
  • KickFromVoice removes seats only through a connection that holds the seat or through a pending grace. A seat in h.voice that no connection and no grace stands for is listed by disconnectForRecheck (it reads h.voice and h.grace) and then not removed. I confirmed it with a scratch test: join a Talk, clear the connection's voice room and its h.rooms entry (the state a half-finished leave would leave), call disconnectForRecheck(nil); the seat is still listed afterwards. kickUserFromChannel already has the missing step (removePresenceUnlessPresent(user, ch, nil) "make sure the seat is gone", internal/ws/perms.go:361-362). Calling removePresence for each seat that was found in h.voice closes it. The state needs an earlier half-done leave to exist, so this is defence in depth, not an observed bug.

S3. GET /api/health now changes outcome during storage blips; the Docker health check is the main caller that sees it

cmd/server/storage.go:100-122, Dockerfile:101-102, docker-compose.yml:187-192.

  • Before: after a healthy start Ready() returned nil for ever, so the container could not turn unhealthy because of storage. After: one HEAD that fails or exceeds 2 s makes /api/health answer 503 for the next 10 s (the probe cache) plus up to 2 s (the handler cache). The image HEALTHCHECK and the compose health check both call /api/health (interval 30 s, retries 3), so storage that is down or slow for roughly 60-90 s marks the app container unhealthy. Docker itself does not restart on that, but anything that watches health does (an auto-heal sidecar, an orchestrator, scripts/backup-common.sh service_start, which waits for healthy, and the self-update page that polls /api/health). doc/deployment.md says "Do not use /api/health as a liveness probe", yet the shipped HEALTHCHECK does exactly that. A short blip (one slow HEAD while SeaweedFS compacts or the NAS is busy) is reported as an outage and clears 10-12 s later; there is no hysteresis.
  • Suggested fix, either or both: count a probe as failed only after two consecutive failures (or keep the last good answer for one TTL when the failure is a timeout), and add a sentence to doc/deployment.md and doc/operations.md that the image health check now turns unhealthy during a longer storage outage and that this is intended. I could not look at the NAS auto-deploy poller (not part of the repository), so whether it reads container health is not checked.
  • Not a problem, checked: the worst case of one /api/health call is the database ping (2 s) plus the storage probe (2 s) = 4 s, under the 5 s Docker timeout, and healthCheck.check serialises callers on its own mutex, so a hung storage costs every waiting caller at most that long, after which they read the cached result. Probe uses HeadBucket, the same call ensureBucket must already have passed before the store exists (internal/s3/s3.go:97), so a store whose credentials forbid HeadBucket never gets the lazy store to the "healthy" state in the first place and cannot be reported down for ever by the probe. MinIO and SeaweedFS answer HEAD on a bucket.

S4. Nothing guarantees that *s3.Client satisfies prober; a mismatch silently disables the new check

cmd/server/storage.go:107-110 (store.(prober); if !ok { return nil }).

If the method signature of (*s3.Client).Probe is ever changed, the assertion fails at run time and Ready goes back to "never reports an outage" with no test failing (the test uses its own fake). Add var _ prober = (*s3.Client)(nil) in cmd/server (or a test that asserts it).

NOTE

N1. Blast radius of "everybody"

PermissionsChanged(nil) that panics disconnects every connection and ends every seat. That is the fail-closed choice and is documented, and the panic is logged only five times a minute per site, so it cannot flood the log. I found no panic source a member can trigger on purpose. If one exists later, an admin's role or overwrite change would be the trigger and the effect a reconnect of the whole community each time the offending member is seated. A per-member guard inside the enforcement loops (disconnect only the member whose step panicked) would shrink this, at the price of more code in perms.go. Not required now.

N2. Documentation is slightly stronger than the code for the HTTP path

doc/operations.md says a panic in the clean-up after a recovered panic ends the process. On a request it does not: the re-panic goes to net/http's recovery and the request answers 500 while the process lives (the change is already committed). With S2 fixed the two cases are closer; otherwise one sentence ("on a request the request fails with 500") is accurate.

N3. h.pending connections are not part of "everybody"

disconnectForRecheck collects users from h.clients, h.voice and h.grace, not from h.pending. DisconnectUser does close pending connections of a user it is given, but a user who has only a pending connection is not in the list. The normal enforcement (filterPermClients) has the same scope, so this is not a regression.

N4. Voice time under a deterministic panic

If the panic in flushVoiceTime repeats (say, a corrupt entry), the seconds are restored and the timer re-armed every voiceTimeFlushEvery: one counted panic per interval for that member and no write, until the member's entry is rewritten. No crash and no double count (the restore runs once; written is set only after the write returned). Between the committed write and written = true there is no code that can panic. In endPresence, a panic inside the locked section after the three deletes and before the seat is removed (h.stage.forget) would, on a retry, take the seat out without the stay's voice time (voiceSince is gone). That is only reachable through the double-panic case already documented.

N5. Web: bounded noticing relies on failing requests and on socket reopen

web/src/stores/serverFeatures.ts, web/src/shared/lib/api.ts:230, web/src/stores/chat/socket.ts:36-39.

  • A healthy server is not polled: the store asks again only while something is reported down (every 30 s), when an api() request answers 503 UNAVAILABLE (at most once per 5 s), and when the socket reopens. XHR uploads go through api() (browserTransport -> xhrUpload), so they are covered. Plain fetch and <img> loads of media that fail with 503 are not, so a member who only reads sees broken pictures and no notice until one of the three triggers happens. The outage is therefore noticed "when it matters (an upload fails)", not within a fixed time. This looks like the intended trade-off; it is stated here so the task's "bounded time" is read the same way.
  • The health check cannot loop on itself: its own 503 is an accepted status, so it never reaches the onUnavailable handlers. A request that never answers is cut after 10 s (AbortSignal.timeout, already used elsewhere in the app, so no new browser requirement) and the timeout is handled like any failed check; during a known outage the 30 s timer is re-armed, otherwise the next call asks again.
  • Listener replacement (stopListening?.()) is only relevant when a second Pinia instance creates the store (tests); in the running app there is one.
  • The 30 s outage timer is not paused for a hidden tab; browsers throttle it, and it runs only while an outage is known.
  • An older server (a 503 with the bare error) still confirms nothing, as before (covered by an existing test).

N6. Review reports in the repository

The added doc/security/reviews/ files contain no local paths, hostnames, addresses or personal data (searched for absolute paths, e-mail addresses, private address ranges). doc/security/reviews/README.md lists the reports of the earlier run only; this PR's own report needs a row once it is added, and rule 9 asks for it in the same pull request or a link. internal/perm/, internal/ws/perms.go and AGENTS.md are CODEOWNERS paths, so this review is required by the rule the PR itself adds; a change after this report needs a new one.

What I ran

All with nice -n 10, one job at a time, on the checkout at head 435356e11 unless noted.

CommandResult
go test -race -count=1 ./internal/ws/ok (46 s)
go test -race -count=1 ./internal/perm/ ./internal/s3/ ./cmd/server/ok, ok, ok
go test -race -count=20 -run 'TestPanicIn|TestGraceTimerPanic' ./internal/ws/ok (no flakes, no race reports)
node scripts/check-doc-links.mjs, node scripts/check-agents-paths.mjsok
scratch worktree: npm ci in web/, then vitest on serverFeaturesOutage.test.ts and connection.test.ts20 passed

Mutations in the scratch worktree (each reverted afterwards):

MutationResult
Web sources serverFeatures.ts, api.ts, chat/socket.ts reverted to v3.0.7, tests kept6 tests fail (reconnect refresh, 503 notice, one check per failed upload, refresh joins a request in flight, timeout, outage recheck after a hung check): the web tests do test the fix
flushVoiceTime: deferred restore removedTestPanicInVoiceTimeTimerKeepsTheSecondsAndTheLock fails
PermissionsChanged: enforceFailClosed removedTestPanicInPermissionEnforcementFailsClosed fails (panic reaches the caller)
Grace timer: onPanic replaced by Abandon, retry removedall grace tests still pass: see S1
Scratch test: seat without connection and grace, then disconnectForRecheck(nil)seat still listed: see S2

The storage test (TestLazyStoreReportsStorageThatDiesAfterTheStartAndItsReturn) was reviewed by reading it; it needs the new probeMu fields to compile, so reverting the fix fails it by construction. I did not run it against a revert.

What I did not check

  • No real S3-compatible store (MinIO, SeaweedFS) was started: the HEAD behaviour is argued from ensureBucket already using the same call.
  • The NAS auto-deploy poller and the setup assistant's use of /api/health beyond what is in the repository (the setup checks call Ready once, 20 s after the start; the metrics endpoint now reports the live value of mnema_storage_up through the same cached call).
  • The whole Go and web suites, the e2e tests, the Docker image and the drills.
  • A panic inside Pion's own goroutines (documented as a process exit, cannot be tested from this code).
  • The SFU: whether SFU.RemovePeer can panic while holding one of its own locks and then deadlock the fail-closed clean-up (the clean-up has no time limit). I read the hub side only.
  • Browser behaviour (no real browser, no audio, no camera); the web checks are jsdom unit tests.
  • The wording of the four new reports in doc/security/reviews/ beyond the privacy search above.

Second round (state 53f7356f0)

Scope: git diff 435356e11 HEAD without the review reports (storage hysteresis, helper split in api.ts and serverFeatures.ts, the grace retry test seam, the per-member fail-closed clean-up). Read-only; mutations ran in a temporary worktree that was removed afterwards.

Verdict: MERGE

S1, S2, S3, S4 and N2 are fixed, and each new test fails without its fix. I found no defect in the fixes; the remarks below are notes.

Status of the earlier findings

FindingStatusEvidence
S1 retry untested, comment wrongFixedTestGraceTimerPanicBeforeTheSeatIsRemovedStillEndsIt injects a panic in the keep check through graceKeep. With the retry removed (onPanic -> Abandon) it fails after 5 s (seat stays listed). The comment now says what the code does.
S2 per-member isolation, seat without connection or graceFixedEach DisconnectUser and each seat runs under its own safego.Run; seats come from h.voice and h.grace and are ended by kickUserFromChannel, which has the "make sure the seat is gone" fallback. Without the per-member Run the isolation test dies with the panic; with KickFromVoice instead of kickUserFromChannel the no-connection test fails.
S3 no hysteresisFixedStorage is reported down after two failed probes in a row (the second after 3 s); a success reports healthy at once. Docs updated (doc/operations.md, doc/deployment.md). With probeFailures = 1 both storage tests fail.
S4 no compile-time checkFixedvar _ prober = (*s3.Client)(nil) in cmd/server/storage.go; cmd/server now imports internal/s3 and builds.
N2 doc on the HTTP pathFixeddoc/operations.md now says: on a timer the process ends, on a request the HTTP layer answers 500 and the process lives.

BLOCKER

None.

SHOULD-FIX

None.

NOTE

N7. DisconnectUser under safego.Locked: no new risk

Same lock (h.mu write lock), same scope, now released on a panic. The only callers in the repository are HTTP handlers and timers (internal/auth/...) and the hub's own disconnectForRecheck; none holds h.mu (it would have deadlocked before as well). Lock order is unchanged: h.mu is not held while c.close() runs. shutdown panicking inside the lock (the isolation test forces it) is now recovered with the lock free.

N8. A member whose own disconnect panics keeps the connection

In that case c.close() of that member is skipped (the panic comes from shutdown, before it). The seat is still ended by the seat loop, so the member holds no Talk seat or media peer, and every right is re-read per request or event; the open socket is therefore not a right the change took away. The test asserts the others are closed and nobody has a seat; it does not assert anything about the broken member's socket, which is the right scope.

N9. kickUserFromChannel for collected seats: no double end, no double event

A seat that is in h.voice and in h.grace is one key in the map, so it is handled once. Inside kickUserFromChannel, leaveVoiceLocked already takes the grace and ends the presence, so for a connected member the following takeGrace is nil and the final removePresenceUnlessPresent is a no-op (wasIn false): one voice leave, one voice_kicked per connection. A seat that only has a grace is ended by removePresence once. A connection that reconnects between the two loops is kicked as well (over-kicks, fail closed). The reason sent is missing_permission even when the real cause was a panic; the old sockets are already shut down and drop it, so only a very fast reconnect could see it.

N10. The test seam graceKeep

Unexported field in package ws, so only code in that package can set it, and nothing outside tests does (grep). It is read once per grace expiry (one nil check per fired timer, not a hot path) and replaces the keep check as a whole, so the test does not exercise seatedLocked; the earlier tests cover that path. -race clean: the field is written before unregister creates the timer.

N11. Hysteresis: behaviour to know

  • Probes only run when somebody asks. The second probe of a blip happens at the next /api/health call after 3 s. With only the 30 s Docker check asking, an outage that began between two checks is reported on the second check, so up to about 60 s after it began, then three failed container checks (about 90 s) make the container unhealthy. That is the intended trade-off of S3 and is within what the docs now say ("about 10-15 s" holds when callers are frequent; with sparse callers it follows their interval).
  • Storage that alternates fail, success, fail, success is never reported, because every success resets the counter. A half-working store is therefore seen as healthy; a hard outage is not affected.
  • Worst case of one /api/health call is unchanged: at most one probe per call (2 s) plus the 2 s database ping, 4 s, under the 5 s Docker timeout; the probe mutex keeps concurrent callers from probing twice.

N12. Web helper split: behaviour unchanged

healthAsker keeps the same state (pending, asking, recheck) in a closure and notifyRefusal picks the same handler set as the two former if lines (rules refusal first; 503 UNAVAILABLE second; a refusal can only be one of them). The existing outage, connection and API tests pass unchanged. eslint on both files and vue-tsc --noEmit are clean. A new Pinia store still registers the failure listener once and replaces the previous one.

What I ran

CommandResult
go test -race -count=1 ./internal/ws/ ./cmd/server/ ./internal/repo/ok, ok, ok (the repo package holds the function size and complexity gates)
go test -race -count=15 -run 'TestGraceTimerPanicBefore|TestFailClosed|TestPanicIn|TestLazyStore' ./internal/ws/ ./cmd/server/ok, no flakes, no race reports
scratch worktree, npm ci, vitest on src/stores/serverFeaturesOutage.test.ts, src/stores/connection.test.ts, src/shared/lib68 files, 1537 tests passed
eslint on serverFeatures.ts, api.ts; vue-tsc --noEmitno findings
Mutations (each reverted): retry removed; per-member Run removed; KickFromVoice instead of kickUserFromChannel; probeFailures = 1the matching new tests fail in all four cases

What I did not check

  • The complete Go and web suites, e2e, the Docker image and the drills.
  • A real S3-compatible store and the NAS auto-deploy poller (outside the repository).
  • Whether SFU.RemovePeer can hold one of its own locks across a panic; the fail-closed clean-up still has no time limit against a hang (carried over from the first round).
  • The four web tests' behaviour under a real browser; only jsdom.

Something wrong on this page? Tell us.