Independent review of pull request 789 (follow-up for 3.0.8)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: an agent that did not write the change. Read-only: nothing in the change was edited; mutations and scratch tests ran in a temporary git worktree that was removed afterwards.
Reviewed: four commits on top of v3.0.7 (c7fafa242), head 435356e11:
the review reports, storage health, the per-member timers, fail-closed
permission enforcement.
Verdict: MERGE AFTER FIXES
No BLOCKER was found. I could not construct a step-by-step scenario in which the change loses a seat, leaks a lock, double-counts voice time or leaves a member with a right a change took away, short of a second panic inside the clean-up (which the change documents as ending the process). The four SHOULD-FIX items are small, and two of them (S1, S4) are untested or silently-disabled code paths that are cheap to close before merge.
BLOCKER
None.
SHOULD-FIX
S1. The grace timer's "end the presence again" branch is untested and its comment is inaccurate
internal/ws/voice_presence.go:121-136 (claimed, onPanic).
- The comment says only a panic before the seat was removed makes
onPanicend the presence again. The code cannot tell the two cases apart: onceclaimedis set,onPanicrunsend()after any panic, including a panic in a listener after the seat is gone. That secondendPresenceis then a no-op (wasInfalse), so it is harmless, but the comment describes something the code does not check. - Evidence that the branch is untested: in a scratch copy I replaced
onPanicbysafego.Abandonand deleted the retry.TestPanicInOneMembersGraceTimerLeavesTheHubConsistent,TestGraceTimerPanicStillTellsTheOthersand the other new tests still passed. Both tests panic inh.Presence, which runs after the seat was removed, so the retry never matters in them. - What a panic before removal could be is thin: inside the locked section only
seatedLockedand map operations run. If one of them did panic deterministically, the retry panics the same way and, as documented, ends the process, so the branch adds little. Either test it with an injected panic in thekeepfunction (a nil entry inh.rooms[ch]makesseatedLockedpanic), or remove the branch and say in the comment that a panic after the removal is abandoned and one before it is not expected. - Smaller consequence of the same code: a retry runs
endPresencea second time. If the member left and a new grace for the same key started in the few microseconds between the two calls, the second call would end that newer seat early (keeponly looks at connections, not at a newer grace). The window is negligible; it is mentioned so the retry is not mistaken for free.
S2. The fail-closed clean-up is not isolated per member, and it does not remove a seat that has no connection and no grace
internal/ws/voice_kick.go:191-217 (disconnectForRecheck).
- The seat loop is guarded per member (
ws_perm_enforce_seat,Abandon), the disconnect loop is not:for id := range users { h.DisconnectUser(id) }runs unguarded. A panic in one member'sDisconnectUser(c.close()->conn.Close()) skips the disconnects of every member after them and the whole seat loop. On the HTTP pathsafego.handlere-panics (panic(again)) into net/http's recovery (internal/httpx/middleware.go:61), so the process continues with some members not re-checked. On timers the process exits. Same guard as for the seats would make the clean-up fail closed per member. KickFromVoiceremoves seats only through a connection that holds the seat or through a pending grace. A seat inh.voicethat no connection and no grace stands for is listed bydisconnectForRecheck(it readsh.voiceandh.grace) and then not removed. I confirmed it with a scratch test: join a Talk, clear the connection's voice room and itsh.roomsentry (the state a half-finished leave would leave), calldisconnectForRecheck(nil); the seat is still listed afterwards.kickUserFromChannelalready has the missing step (removePresenceUnlessPresent(user, ch, nil)"make sure the seat is gone",internal/ws/perms.go:361-362). CallingremovePresencefor each seat that was found inh.voicecloses it. The state needs an earlier half-done leave to exist, so this is defence in depth, not an observed bug.
S3. GET /api/health now changes outcome during storage blips; the Docker health check is the main caller that sees it
cmd/server/storage.go:100-122, Dockerfile:101-102, docker-compose.yml:187-192.
- Before: after a healthy start
Ready()returned nil for ever, so the container could not turn unhealthy because of storage. After: one HEAD that fails or exceeds 2 s makes/api/healthanswer 503 for the next 10 s (the probe cache) plus up to 2 s (the handler cache). The imageHEALTHCHECKand the compose health check both call/api/health(interval 30 s, retries 3), so storage that is down or slow for roughly 60-90 s marks theappcontainerunhealthy. Docker itself does not restart on that, but anything that watches health does (an auto-heal sidecar, an orchestrator,scripts/backup-common.shservice_start, which waits forhealthy, and the self-update page that polls/api/health).doc/deployment.mdsays "Do not use/api/healthas a liveness probe", yet the shippedHEALTHCHECKdoes exactly that. A short blip (one slow HEAD while SeaweedFS compacts or the NAS is busy) is reported as an outage and clears 10-12 s later; there is no hysteresis. - Suggested fix, either or both: count a probe as failed only after two
consecutive failures (or keep the last good answer for one TTL when the
failure is a timeout), and add a sentence to
doc/deployment.mdanddoc/operations.mdthat the image health check now turnsunhealthyduring a longer storage outage and that this is intended. I could not look at the NAS auto-deploy poller (not part of the repository), so whether it reads container health is not checked. - Not a problem, checked: the worst case of one
/api/healthcall is the database ping (2 s) plus the storage probe (2 s) = 4 s, under the 5 s Docker timeout, andhealthCheck.checkserialises callers on its own mutex, so a hung storage costs every waiting caller at most that long, after which they read the cached result.ProbeusesHeadBucket, the same callensureBucketmust already have passed before the store exists (internal/s3/s3.go:97), so a store whose credentials forbidHeadBucketnever gets the lazy store to the "healthy" state in the first place and cannot be reported down for ever by the probe. MinIO and SeaweedFS answer HEAD on a bucket.
S4. Nothing guarantees that *s3.Client satisfies prober; a mismatch silently disables the new check
cmd/server/storage.go:107-110 (store.(prober); if !ok { return nil }).
If the method signature of (*s3.Client).Probe is ever changed, the assertion
fails at run time and Ready goes back to "never reports an outage" with no
test failing (the test uses its own fake). Add
var _ prober = (*s3.Client)(nil) in cmd/server (or a test that asserts it).
NOTE
N1. Blast radius of "everybody"
PermissionsChanged(nil) that panics disconnects every connection and ends
every seat. That is the fail-closed choice and is documented, and the panic is
logged only five times a minute per site, so it cannot flood the log. I found
no panic source a member can trigger on purpose. If one exists later, an
admin's role or overwrite change would be the trigger and the effect a
reconnect of the whole community each time the offending member is seated.
A per-member guard inside the enforcement loops (disconnect only the member
whose step panicked) would shrink this, at the price of more code in
perms.go. Not required now.
N2. Documentation is slightly stronger than the code for the HTTP path
doc/operations.md says a panic in the clean-up after a recovered panic ends
the process. On a request it does not: the re-panic goes to net/http's
recovery and the request answers 500 while the process lives (the change is
already committed). With S2 fixed the two cases are closer; otherwise one
sentence ("on a request the request fails with 500") is accurate.
N3. h.pending connections are not part of "everybody"
disconnectForRecheck collects users from h.clients, h.voice and
h.grace, not from h.pending. DisconnectUser does close pending
connections of a user it is given, but a user who has only a pending
connection is not in the list. The normal enforcement (filterPermClients)
has the same scope, so this is not a regression.
N4. Voice time under a deterministic panic
If the panic in flushVoiceTime repeats (say, a corrupt entry), the seconds
are restored and the timer re-armed every voiceTimeFlushEvery: one counted
panic per interval for that member and no write, until the member's entry is
rewritten. No crash and no double count (the restore runs once; written is
set only after the write returned). Between the committed write and
written = true there is no code that can panic. In endPresence, a panic
inside the locked section after the three deletes and before the seat is
removed (h.stage.forget) would, on a retry, take the seat out without the
stay's voice time (voiceSince is gone). That is only reachable through the
double-panic case already documented.
N5. Web: bounded noticing relies on failing requests and on socket reopen
web/src/stores/serverFeatures.ts, web/src/shared/lib/api.ts:230,
web/src/stores/chat/socket.ts:36-39.
- A healthy server is not polled: the store asks again only while something
is reported down (every 30 s), when an
api()request answers503 UNAVAILABLE(at most once per 5 s), and when the socket reopens. XHR uploads go throughapi()(browserTransport->xhrUpload), so they are covered. Plainfetchand<img>loads of media that fail with 503 are not, so a member who only reads sees broken pictures and no notice until one of the three triggers happens. The outage is therefore noticed "when it matters (an upload fails)", not within a fixed time. This looks like the intended trade-off; it is stated here so the task's "bounded time" is read the same way. - The health check cannot loop on itself: its own 503 is an accepted status,
so it never reaches the
onUnavailablehandlers. A request that never answers is cut after 10 s (AbortSignal.timeout, already used elsewhere in the app, so no new browser requirement) and the timeout is handled like any failed check; during a known outage the 30 s timer is re-armed, otherwise the next call asks again. - Listener replacement (
stopListening?.()) is only relevant when a second Pinia instance creates the store (tests); in the running app there is one. - The 30 s outage timer is not paused for a hidden tab; browsers throttle it, and it runs only while an outage is known.
- An older server (a 503 with the bare error) still confirms nothing, as before (covered by an existing test).
N6. Review reports in the repository
The added doc/security/reviews/ files contain no local paths, hostnames,
addresses or personal data (searched for absolute paths, e-mail addresses,
private address ranges). doc/security/reviews/README.md lists the reports of
the earlier run only; this PR's own report needs a row once it is added, and
rule 9 asks for it in the same pull request or a link. internal/perm/,
internal/ws/perms.go and AGENTS.md are CODEOWNERS paths, so this review is
required by the rule the PR itself adds; a change after this report needs a
new one.
What I ran
All with nice -n 10, one job at a time, on the checkout at head 435356e11
unless noted.
| Command | Result |
|---|---|
go test -race -count=1 ./internal/ws/ | ok (46 s) |
go test -race -count=1 ./internal/perm/ ./internal/s3/ ./cmd/server/ | ok, ok, ok |
go test -race -count=20 -run 'TestPanicIn|TestGraceTimerPanic' ./internal/ws/ | ok (no flakes, no race reports) |
node scripts/check-doc-links.mjs, node scripts/check-agents-paths.mjs | ok |
scratch worktree: npm ci in web/, then vitest on serverFeaturesOutage.test.ts and connection.test.ts | 20 passed |
Mutations in the scratch worktree (each reverted afterwards):
| Mutation | Result |
|---|---|
Web sources serverFeatures.ts, api.ts, chat/socket.ts reverted to v3.0.7, tests kept | 6 tests fail (reconnect refresh, 503 notice, one check per failed upload, refresh joins a request in flight, timeout, outage recheck after a hung check): the web tests do test the fix |
flushVoiceTime: deferred restore removed | TestPanicInVoiceTimeTimerKeepsTheSecondsAndTheLock fails |
PermissionsChanged: enforceFailClosed removed | TestPanicInPermissionEnforcementFailsClosed fails (panic reaches the caller) |
Grace timer: onPanic replaced by Abandon, retry removed | all grace tests still pass: see S1 |
Scratch test: seat without connection and grace, then disconnectForRecheck(nil) | seat still listed: see S2 |
The storage test (TestLazyStoreReportsStorageThatDiesAfterTheStartAndItsReturn)
was reviewed by reading it; it needs the new probeMu fields to compile, so
reverting the fix fails it by construction. I did not run it against a revert.
What I did not check
- No real S3-compatible store (MinIO, SeaweedFS) was started: the HEAD
behaviour is argued from
ensureBucketalready using the same call. - The NAS auto-deploy poller and the setup assistant's use of
/api/healthbeyond what is in the repository (the setup checks callReadyonce, 20 s after the start; the metrics endpoint now reports the live value ofmnema_storage_upthrough the same cached call). - The whole Go and web suites, the e2e tests, the Docker image and the drills.
- A panic inside Pion's own goroutines (documented as a process exit, cannot be tested from this code).
- The SFU: whether
SFU.RemovePeercan panic while holding one of its own locks and then deadlock the fail-closed clean-up (the clean-up has no time limit). I read the hub side only. - Browser behaviour (no real browser, no audio, no camera); the web checks are jsdom unit tests.
- The wording of the four new reports in
doc/security/reviews/beyond the privacy search above.
Second round (state 53f7356f0)
Scope: git diff 435356e11 HEAD without the review reports (storage
hysteresis, helper split in api.ts and serverFeatures.ts, the grace retry
test seam, the per-member fail-closed clean-up). Read-only; mutations ran in a
temporary worktree that was removed afterwards.
Verdict: MERGE
S1, S2, S3, S4 and N2 are fixed, and each new test fails without its fix. I found no defect in the fixes; the remarks below are notes.
Status of the earlier findings
| Finding | Status | Evidence |
|---|---|---|
| S1 retry untested, comment wrong | Fixed | TestGraceTimerPanicBeforeTheSeatIsRemovedStillEndsIt injects a panic in the keep check through graceKeep. With the retry removed (onPanic -> Abandon) it fails after 5 s (seat stays listed). The comment now says what the code does. |
| S2 per-member isolation, seat without connection or grace | Fixed | Each DisconnectUser and each seat runs under its own safego.Run; seats come from h.voice and h.grace and are ended by kickUserFromChannel, which has the "make sure the seat is gone" fallback. Without the per-member Run the isolation test dies with the panic; with KickFromVoice instead of kickUserFromChannel the no-connection test fails. |
| S3 no hysteresis | Fixed | Storage is reported down after two failed probes in a row (the second after 3 s); a success reports healthy at once. Docs updated (doc/operations.md, doc/deployment.md). With probeFailures = 1 both storage tests fail. |
| S4 no compile-time check | Fixed | var _ prober = (*s3.Client)(nil) in cmd/server/storage.go; cmd/server now imports internal/s3 and builds. |
| N2 doc on the HTTP path | Fixed | doc/operations.md now says: on a timer the process ends, on a request the HTTP layer answers 500 and the process lives. |
BLOCKER
None.
SHOULD-FIX
None.
NOTE
N7. DisconnectUser under safego.Locked: no new risk
Same lock (h.mu write lock), same scope, now released on a panic. The only
callers in the repository are HTTP handlers and timers (internal/auth/...)
and the hub's own disconnectForRecheck; none holds h.mu (it would have
deadlocked before as well). Lock order is unchanged: h.mu is not held while
c.close() runs. shutdown panicking inside the lock (the isolation test
forces it) is now recovered with the lock free.
N8. A member whose own disconnect panics keeps the connection
In that case c.close() of that member is skipped (the panic comes from
shutdown, before it). The seat is still ended by the seat loop, so the
member holds no Talk seat or media peer, and every right is re-read per
request or event; the open socket is therefore not a right the change took
away. The test asserts the others are closed and nobody has a seat; it does not
assert anything about the broken member's socket, which is the right scope.
N9. kickUserFromChannel for collected seats: no double end, no double event
A seat that is in h.voice and in h.grace is one key in the map, so it is
handled once. Inside kickUserFromChannel, leaveVoiceLocked already takes
the grace and ends the presence, so for a connected member the following
takeGrace is nil and the final removePresenceUnlessPresent is a no-op
(wasIn false): one voice leave, one voice_kicked per connection. A seat
that only has a grace is ended by removePresence once. A connection that
reconnects between the two loops is kicked as well (over-kicks, fail closed).
The reason sent is missing_permission even when the real cause was a
panic; the old sockets are already shut down and drop it, so only a very fast
reconnect could see it.
N10. The test seam graceKeep
Unexported field in package ws, so only code in that package can set it, and
nothing outside tests does (grep). It is read once per grace expiry (one nil
check per fired timer, not a hot path) and replaces the keep check as a whole,
so the test does not exercise seatedLocked; the earlier tests cover that
path. -race clean: the field is written before unregister creates the
timer.
N11. Hysteresis: behaviour to know
- Probes only run when somebody asks. The second probe of a blip happens at
the next
/api/healthcall after 3 s. With only the 30 s Docker check asking, an outage that began between two checks is reported on the second check, so up to about 60 s after it began, then three failed container checks (about 90 s) make the container unhealthy. That is the intended trade-off of S3 and is within what the docs now say ("about 10-15 s" holds when callers are frequent; with sparse callers it follows their interval). - Storage that alternates fail, success, fail, success is never reported, because every success resets the counter. A half-working store is therefore seen as healthy; a hard outage is not affected.
- Worst case of one
/api/healthcall is unchanged: at most one probe per call (2 s) plus the 2 s database ping, 4 s, under the 5 s Docker timeout; the probe mutex keeps concurrent callers from probing twice.
N12. Web helper split: behaviour unchanged
healthAsker keeps the same state (pending, asking, recheck) in a
closure and notifyRefusal picks the same handler set as the two former if
lines (rules refusal first; 503 UNAVAILABLE second; a refusal can only be
one of them). The existing outage, connection and API tests pass unchanged.
eslint on both files and vue-tsc --noEmit are clean. A new Pinia store
still registers the failure listener once and replaces the previous one.
What I ran
| Command | Result |
|---|---|
go test -race -count=1 ./internal/ws/ ./cmd/server/ ./internal/repo/ | ok, ok, ok (the repo package holds the function size and complexity gates) |
go test -race -count=15 -run 'TestGraceTimerPanicBefore|TestFailClosed|TestPanicIn|TestLazyStore' ./internal/ws/ ./cmd/server/ | ok, no flakes, no race reports |
scratch worktree, npm ci, vitest on src/stores/serverFeaturesOutage.test.ts, src/stores/connection.test.ts, src/shared/lib | 68 files, 1537 tests passed |
eslint on serverFeatures.ts, api.ts; vue-tsc --noEmit | no findings |
Mutations (each reverted): retry removed; per-member Run removed; KickFromVoice instead of kickUserFromChannel; probeFailures = 1 | the matching new tests fail in all four cases |
What I did not check
- The complete Go and web suites, e2e, the Docker image and the drills.
- A real S3-compatible store and the NAS auto-deploy poller (outside the repository).
- Whether
SFU.RemovePeercan hold one of its own locks across a panic; the fail-closed clean-up still has no time limit against a hang (carried over from the first round). - The four web tests' behaviour under a real browser; only jsdom.