Review of PR #917: "perf(ws): spread reconnect storms and cut their per-reconnect cost"

From mnema-talk v3.3.13, rendered on this site.

Verdict: MERGE AFTER FIXES. An independent reviewer read the diff and its tests. All findings below are resolved on the branch or accepted as they are.

An earlier review round asked for a fix to close frames lost in a held batch: batchConn.Close now sends the batch before it closes the connection (TestBatchConnCloseSendsTheHeldBatch).

M1 (Medium): a hung database could report healthy

pingOrBusy treated a timed-out ping as "busy, not down" when the pool's handed-out minus held count grew. That count also grows when pgxpool destroys connections whose requests were cancelled against a hung or black-holed database. In that case /api/health answered 200 for a database that answered nothing.

Resolution: db.Pool counts queries that finish without an error (QueriesDone, from a pgx QueryTracer, TraceQueryEnd with Err == nil). A ping timeout counts as busy only if that counter grew by at least 2 during the ping.

  • TestPingOrBusy has the case "hung: connections destroyed by cancelled requests, no successful query", which must report down.
  • TestQueriesDoneCountsOnlyQueriesThatFinished (integration) cancels queries against pg_sleep. It shows the old pool count moving while QueriesDone stays at 0.

L1 (Low): Close's short wait could be stretched to writeWait

gorilla calls SetWriteDeadline(now + writeWait) before every write. If that happened while Close was waiting on a stuck write, the 250 ms bound became 10 s, and the hub closes clients one after another (shutdown, sign-out, kicks).

Resolution: once Close begins, batchConn records its deadline, and SetWriteDeadline/SetDeadline never move the write deadline past it. TestBatchConnCloseKeepsItsDeadlineAgainstALaterOne keeps setting a 10 s deadline while Close runs; it fails on the previous code and passes now.

L2 (Low): the shared member read carried one request's context values

The shared read of GET /api/members ran on context.WithoutCancel of the request that started it. Its user, request id and address then appeared in the logs and traces of a read that serves everyone.

Resolution: the shared read starts from context.Background() with its own timeout and carries no request values; the code comment says so.

I1 (Info): the first backoff window is narrow

The client's first retry waits 0.5 s to 1 s, so a fast restart still brings most clients back within half a second.

Accepted. Measured: with about 0.5 s of downtime, all 1,000 sessions were back in 4.4 to 4.6 s (it was 6.6 to 6.7 s). Widening the first window would delay every ordinary reconnect, a network blip included. Later windows spread the herd: with 8 s down, the upgrade p99 fell from 311 to 985 ms to 10 to 16 ms.

I2 (Info): a batch copies payloads that are shared

batchConn copies each frame into its own buffer, including the encoded broadcast payloads that all connections share.

Accepted. A batch is bounded: at most 64 frames or 32 KiB. A buffer larger than 4 KiB is dropped after the burst, so an idle connection keeps at most 4 KiB. The copy replaces one system call per frame, which cost far more.

Something wrong on this page? Tell us.