Review of PR #917: "perf(ws): spread reconnect storms and cut their per-reconnect cost"
From mnema-talk v3.3.13, rendered on this site.
Verdict: MERGE AFTER FIXES. An independent reviewer read the diff and its tests. All findings below are resolved on the branch or accepted as they are.
An earlier review round asked for a fix to close frames lost in a held batch:
batchConn.Close now sends the batch before it closes the connection
(TestBatchConnCloseSendsTheHeldBatch).
M1 (Medium): a hung database could report healthy
pingOrBusy treated a timed-out ping as "busy, not down" when the pool's
handed-out minus held count grew. That count also grows when pgxpool destroys
connections whose requests were cancelled against a hung or black-holed
database. In that case /api/health answered 200 for a database that answered
nothing.
Resolution: db.Pool counts queries that finish without an error
(QueriesDone, from a pgx QueryTracer, TraceQueryEnd with Err == nil).
A ping timeout counts as busy only if that counter grew by at least 2 during
the ping.
TestPingOrBusyhas the case "hung: connections destroyed by cancelled requests, no successful query", which must report down.TestQueriesDoneCountsOnlyQueriesThatFinished(integration) cancels queries againstpg_sleep. It shows the old pool count moving whileQueriesDonestays at 0.
L1 (Low): Close's short wait could be stretched to writeWait
gorilla calls SetWriteDeadline(now + writeWait) before every write. If that
happened while Close was waiting on a stuck write, the 250 ms bound became
10 s, and the hub closes clients one after another (shutdown, sign-out, kicks).
Resolution: once Close begins, batchConn records its deadline, and
SetWriteDeadline/SetDeadline never move the write deadline past it.
TestBatchConnCloseKeepsItsDeadlineAgainstALaterOne keeps setting a 10 s
deadline while Close runs; it fails on the previous code and passes now.
L2 (Low): the shared member read carried one request's context values
The shared read of GET /api/members ran on context.WithoutCancel of the
request that started it. Its user, request id and address then appeared in the
logs and traces of a read that serves everyone.
Resolution: the shared read starts from context.Background() with its own
timeout and carries no request values; the code comment says so.
I1 (Info): the first backoff window is narrow
The client's first retry waits 0.5 s to 1 s, so a fast restart still brings most clients back within half a second.
Accepted. Measured: with about 0.5 s of downtime, all 1,000 sessions were back in 4.4 to 4.6 s (it was 6.6 to 6.7 s). Widening the first window would delay every ordinary reconnect, a network blip included. Later windows spread the herd: with 8 s down, the upgrade p99 fell from 311 to 985 ms to 10 to 16 ms.
I2 (Info): a batch copies payloads that are shared
batchConn copies each frame into its own buffer, including the encoded
broadcast payloads that all connections share.
Accepted. A batch is bounded: at most 64 frames or 32 KiB. A buffer larger than 4 KiB is dropped after the burst, so an idle connection keeps at most 4 KiB. The copy replaces one system call per frame, which cost far more.