Independent review of pull request 817 (nightly sweeps)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: an agent that did not write the change. Read-only: nothing in the
change was edited. The probes ran as a scratch test file inside
internal/automod that was deleted afterwards; the check without the fix put
the origin/main version of internal/automod/linkhost.go in place for one
test run and then restored it with git checkout HEAD --. The worktree was
clean afterwards. No real instance was contacted.
Reviewed: one commit, head dc6f83774 ("link rules apply to hosts with a bare
ACE label as well"), on merge base 502b1056b. origin/main is one commit
further (0945ee104, #816); that commit touches none of the files below.
internal/automod/linkhost.go(read in full)internal/automod/linkscan_test.go(read in full)internal/automod/testdata/link_parity.json(the added vectors)web/src/shared/lib/markdown/linkParity.cases.fixture.ts(read in full).github/workflows/visibility-sweep.yml(read in full)
Read around it, in full: internal/automod/linkscan.go, linkends.go,
linkParity.fixture.ts, linkParity.test.ts, linkParity.sweep.test.ts,
linkParity.current.test.ts. In part: compile.go (forbidsLinks,
coveredBy), rule.go (cleanDomain, hostOf), linkcover.go (the work
budget), the two integration tests the workflow runs, and
golang.org/x/net@v0.62.0/idna (idna.go process, punycode.go decode).
Verdict: MERGEABLE
No BLOCKER, no SHOULD-FIX. Two informational notes below.
Root cause
golang.org/x/net/idna (v0.62.0, idna.go:393-411) strips the ACE prefix and
Punycode-decodes the rest of the label. decode("") returns "", nil
(punycode.go:35-37), and the "decoded label is ASCII" error is raised only
for len(u) > 0, so a label that is exactly xn-- (after mapping) is replaced
by an empty label with no error. With VerifyDNSLength(false), as the URL
Standard requires, nothing reports the empty label either. readDomain then
read the host without that label and marked it exact. The URL parser in
Node 26 keeps an ASCII xn-- label as written. So the server judged a host
that differs from the one the client opens.
The fix is in the right place. readDomain is the single host reader both for
messages (linkends.go:41) and for configured domains (rule.go:295).
keepEmptyACE puts the bare ACE label back and marks the reading as not
exact. With that, link rules apply to hosts with a bare ACE label as well: an
allow list refuses them, as it refuses every reading it cannot take as exact,
and a block list judges the host the client opens. A configured domain with
such a label is no longer accepted, because cleanDomain requires an exact
reading.
What I checked
- Detection condition (
linkhost.go:114). An emptied label always shows in Go's output as an empty host, a leading or trailing dot, or... Hosts without these return early, so ordinary hosts take no extra work. A trailing dot or..without an ACE label goes through the per-rune mapping but comes back unchanged and exact (allowed.example.,a..allowed.example,sub.allowed.example.). - Per-rune mapping (
linkhost.go:137-148). The comment says composition never yields an ASCIIx,nor-. That holds: NFC never composes into ASCII. Probed inputs: ASCII upper and mixed case (XN--,Xn--,xN--), percent-encoded (%78n--), full-width letters and hyphens, full-width, ideographic and half-width ideographic full stops as separators, and ignorable code points mapped to nothing (U+00AD in three positions, U+034F, U+FE00, U+200B). Each was read with the bare label kept and as not exact. U+200D (kept by non-transitional processing), U+2024, U+FE52 and U+2488 turn the label into another label. Go then encodes it normally and reports an error, so the reading is not exact either. - Several labels, positions, trailing dots:
xn--.xn--,xn--.xn--.allowed.example,.xn--,..xn--,.xn--.allowed.example,xn--..allowed.example,xn--.,xn--..,allowed.example.xn--.,allowed.example.xn--.., mixed with non-ASCII labels (café.xn--,xn--.é.allowed.example,xn--caf-dma.xn--). All of them kept every label and were not exact. For the ASCII forms the host matched what Node 26'snew URL()reports, apart from the trailing dot, which the code trims for every host. - Hosts that come out empty: a lone
xn--(any case, with user info and a port) now reads as hostxn--, neither exact nor rejected. Node 26 opens it. Before the fix it was rejected and could be left out when another reading existed. - Label-count mismatch (
linkhost.go:122-124). When the per-rune mapping and Go's output disagree on the number of labels, Go's host is returned and marked as not exact. This is the conservative direction. None of the inputs above reached this branch. - IPv4 interplay:
xn--.1andxn--.0x7fare rejected (they end in a number and are no address), and Node 26 rejects them too.1.2.3.xn--is a domain and not exact. - No false alarms on allowed links. With an allow list of
allowed.example, these still pass:https://allowed.example/x,https://sub.allowed.example./x,see https://allowed.example.,**https://allowed.example**,https://xn--caf-dma.allowed.example/x,https://café.allowed.example/xand[a](https://allowed.example/x). ThePASSINGset of the parity fixture stays green as well. - Block lists. Before and after the fix, a host whose ACE label decodes
to ASCII (for example
xn--a-) reads as the decoded label and not as exact. Configured domains must be exact, so they contain no such label, and the labels a listed domain covers are left unchanged by Go's processing. A host the client opens under a listed domain therefore stays covered. Nothing to fix here. - Cost. A 3.9 kB message full of tokens with emptied labels takes about
0.20 ms against 0.13 ms for the same message without them (Apple M5,
go test -bench). The work budget is unchanged and charges the reading as before. - The new tests fail without the fix. With the
origin/mainversion oflinkhost.go,TestAnEmptyACELabelStaysInTheHostfails on all 9 host cases and all 4 messages.TestLinkParityVectorsfails on all 5 new vectors. The exact-host half of the new test passes on both versions, as it should. With the fix, everything passes. - Fixture and vectors agree. The five messages added to
AGREEDare inlink_parity.jsonin the orderlinkParity.test.tsexpects (AGREED, thenPASSING, then the sample). The server readings recorded there match what the fixed code produces. - Workflow (
visibility-sweep.yml:43-89). The matrix is nowparity × sweep. Eachincludeentry matches one existingsweepvalue and only addstest, so this gives four jobs, each running exactly one test. I confirmed withgo test -tags=integration -listthat both names exist. The-runpattern is anchored, and the test name comes in throughenv, not interpolated into the script.VIS_SWEEP=1, the seed formulaGITHUB_RUN_NUMBER * 2 + PARITY, the sizes (deep canary sweep, 400 communities) and both tests' assertions are unchanged. Each test gets the same seed as before, so no seed is dropped. Thego testtimeout goes from 40 to 50 minutes for one test instead of two, inside the unchanged 60-minute job timeout.fail-fast: falsestill holds. Nothing in the gate is weakened. The four jobs each start their own PostgreSQL service. The two tests do not depend on each other's data.
What I ran
| Command | Result |
|---|---|
go test -count=1 ./internal/automod | ok |
same, with linkhost.go from origin/main | FAIL: TestLinkParityVectors (5 vectors), TestAnEmptyACELabelStaysInTheHost (13 cases); restored afterwards |
scratch zz_review_test.go (46 authorities, 11 messages, two benchmarks) | results as in checks 1 to 9; file deleted |
Node 26.13.1 new URL() on 29 of the same authorities | ASCII forms kept as written; non-ASCII mapped forms fail to parse |
PATH=…/v26.13.1/…:$PATH npx vitest run src/shared/lib/markdown/ (in web/) | 13 files passed, 1 skipped (the deep sweep, needs LINK_SWEEP); 433 tests passed, 4 skipped |
actionlint .github/workflows/visibility-sweep.yml | no findings |
go vet -tags=integration ./internal/server | ok |
go test -tags=integration -list '^(TestNoCanary…|TestRandomCommunities…)$' ./internal/server | both names listed |
What I did not check
- The deep link sweep itself (
LINK_SWEEP=1, 100,000 messages per shard) and the nightly visibility jobs were not run. Their run time on the CI runners under the new matrix is taken from the workflow comment, not measured. - Browsers other than Node 26's URL parser (Chromium, WebKit, Firefox) were not compared on these hosts. Where they reject a non-ASCII mapped form, the server's reading as not exact is stricter, not looser.
- No full UTS #46 conformance run of the per-rune mapping. The probes cover the classes listed above.
Findings
I1. The generator has no host with a bare ACE label (informational)
web/src/shared/lib/markdown/linkParity.fixture.ts:87 (HOSTS). The nightly
run found this class by chance, by splicing characters into
xn--caf-dma.example. Adding a host such as xn--.allowed.example (and one
with the label last) to HOSTS would make the sweep cover the class on
purpose. Changing HOSTS changes the 500-message sample, so the vector file
would have to be written again. This is optional and can go in a later change.
I2. Non-ASCII mapped forms read as not exact where Node 26 rejects them (informational)
internal/automod/linkhost.go:92-94. A full-width or ignorable-character
spelling of the bare ACE label (xn--.allowed.example,
xn--.allowed.example) fails to parse in Node 26, so the client links
nothing from it. The server now reads it as not exact, and an allow list
refuses the message. This is stricter than the client, which is the intended
direction ("never as exact"). Ordinary allowed links are not affected (check 7).
No change needed.