Fix review: H-L3-01..04 (automod link rules against the client renderer)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: independent agent, phase 3, read-only. Fix branch msch128/fix-h-l3-links, fix commit adc32bacc (on the two rebased test commits 78814b9e2 and 1ae11aeab). Reviewed: the diff of adc32bacc only, with the surrounding production code read in full: internal/automod/linkscan.go, linkends.go, linkhost.go, compile.go, rule.go (hostOf, cleanDomain), engine.go (new-account rule); web/src/shared/lib/markdown/inline.ts, blocks.ts, render.ts, link.ts, linkAuthority.ts, plain.ts, mentions.ts, markdown.ts; LinkPreviewCard.vue; the test files and fixtures of the fix; .github/workflows/sim-sweep.yml; doc/moderation.md (Links row). Findings: H-L3-01..04. Phase 2: repro-H-L3-01-04, refute-H-L3-01-04 (mechanisms M1..M5).

Verdict

DO NOT MERGE as it stands (one BLOCKER). The fix removes M1..M5 for every variant the finder and the refuter list, and none of my 60,000 own fuzz messages or 1,500,000 sweep messages on new seeds found a bypass, but the client has one more place where it ends a link host that the server does not model, and through it an ordinary member gets a link to a listed (block list) or unlisted (allow list) host past the rule, in the current bundle and in older ones, in Chromium. After the blocker is fixed: MERGE AFTER FIXES (the SHOULD-FIX items).

FindingCause removed?Verdict
H-L3-01 user info behind ' and NFKC stopsyes, all listed forms (M1, M2)covered, but see BLOCKER (same property)
H-L3-02 client ends the link earlieryes for every listed end (M3, M4); one client end missing (BLOCKER)DO NOT MERGE
H-L3-03 nested linkyes (M5)covered
H-L3-04 mark kept in the hostyes (M1, URL Standard host parser)covered

The four share one property and one fix commit; the verdict is one for the commit.

BLOCKER

B1. The masked-link destination length cap is a client end the server does not model. web/src/shared/lib/markdown/inline.ts:227 (destination): the client reads the target of text (and text) for at most MAX_LABEL * 4 = 2000 UTF-16 code units and then cuts it; when what follows the cut is a valid tail (a title in quotes or parentheses and )), the link is rendered with the cut URL. If the cut falls inside the authority, the client opens the host up to the cut. internal/automod/linkends.go:62-80 (clientEnds) knows the framed ends at each ) and at the end of the authority, but not this one, so the server reads only longer hosts. When those longer readings end in a character no DNS label has, the allow list ignores them (compile.go:136, the reachable rule), and the block list does not find the listed domain in them (they are not a subdomain of it). Measured:

  • Server, real message route (temporary integration test in internal/server, same fixture as the fix's tests, ordinary 30-day member): a message of 2008 characters, ASCII only, posted under allow_list allowed.example and under block_list other.example: 201 in both.
  • Client, current bundle (renderMarkdown in Node 24 and the same href in headless Chromium 158): an <a> whose host is a subdomain of other.example. With the older bundle (user info not refused) a second form with a user name also works. Block-all and the new-account rule still refuse (any link counts).
  • The current bundle's plainAuthority does not stop it: the padding that brings the authority to the cap can be made of percent-escaped characters that UTS #46 maps to nothing, and linkAuthority.ts:14 checks the raw text for Cc/Cf, not the decoded host.

Neither the hand-written cases nor the generator (linkParity.fixture.ts WRAPS: destinations are always short) reach the cap, which is why 12,000,000 generated messages found nothing. The exact message is withheld from this public report and was given to the coordinator.

Fix direction (server side is required, older bundles have the same cap; inline.ts is unchanged by this commit): when a token is framed by ]( or ](<, add an end at the position 2000 UTF-16 code units after the destination start when it falls inside the authority (or simply treat a framed authority longer than that as opaque, which no ordinary link reaches). Optionally also make the client drop a destination that hits the cap instead of cutting it. Add a long-destination case to the cases fixture and the generator.

SHOULD-FIX

S1. The cost claim is wrong by an order of magnitude; work per message is quadratic. The commit message says hostile 4000-character messages take at most 150 us. My own worst cases (links() directly, Apple M-series, -benchtime 100x), new against old code (adc32bacc1):

input (at most 4000 characters)newold
one authority of 3960 non-ASCII letters, 15 * closers at its end (16 IDNA readings of the whole host)1.59 ms0.25 ms
280 nested https://a, then 1480 trailing . (each nested scheme trims the shared tail again)1.07 ms0.06 ms
40 nested links, each a 40-letter non-ASCII host with 15 closers0.97 ms0.08 ms
2000-character token of `.,;:!?*_~` runs0.03 ms
the author's own hostile set (BenchmarkMatchesLinks)57-109 us-

There is a bound (at most 16 readings per authority, one shared token end per run of nested schemes), but it is O(schemes x token length) and O(16 x host length x IDNA). Not a blocker at ~1.6 ms per message with the posting rate limits, but: correct the claim, and note that the new-account rule calls links() again (engine.go:238, 246), so a server with the Links rule and the default new-account rule does the work twice. Suggested: compute links(content) once per Check, and trim the shared tail once per token instead of per nested scheme.

S2. The PR-time parity test only renders as the older bundle (linkParity.test.ts:21 mocks plainAuthority to true), but the current bundle is not a subset of the older one. When plainAuthority refuses an outer link (user info), the client goes on scanning and links a nested URL inside it, which the older bundle swallowed in the outer href. The server judges every nested scheme, so it holds today, and the nightly sweep runs LINK_SWEEP_CLIENT=current. But the smoke test should check both renderings (run the AGREED and generated rows twice), otherwise a regression in nested judging that only the current bundle exposes is found a day late.

S3. linkAuthority.ts checks the raw authority, not the host the browser reads. %E2%80%8B (zero width space), %C2%AD (soft hyphen) and every other percent-escaped invisible character pass plainAuthority, the browser decodes them and IDNA maps them away (measured: https://othe%E2%80%8Br.example is linked and opens other.example). It is defence in depth only (the server holds without it), but the stated purpose "none with a control or invisible format character" is not met; decode the authority (or compare against new URL(raw).hostname) before the test. This is also what B1's current-bundle form uses.

Over-blocking (main product risk)

Method: 71 realistic messages of my own (search result URLs, Google/Facebook/YouTube redirect links, web archive and archive.ph URLs, Wikipedia URLs with parentheses, URLs followed by punctuation, inside quotes, after emoji, in <...>, masked links with titles, non-ASCII paths, IDN hosts, lists, quotes, spoilers, code spans and fenced code), judged by the real matchesLinks, rendered by the real client.

  • Allow list of 7 common domains (github.com, wikipedia.org, youtube.com, youtu.be, google.com, reddit.com, stackoverflow.com): the server refuses 21 messages; old code 19. The 2 new refusals are both redirect-style links whose opened host is allowed (youtube.com/redirect?q=https://<unlisted>, translate.google.com/translate?...&u=https://<unlisted>). One more refusal (also in the old code) is a URL inside a fenced code block, which the client does not link. With a wide allow list (20 domains) 4 of 71 are refused, all nested or code.
  • Block list of 6 domains: server refuses 5, old code 1. The 4 new refusals all have a blocked host only inside a query or path (archive.ph/https://<blocked>, youtube.com/redirect?q=https://<blocked>, accounts.google.com/...?continue=https://<blocked-subdomain>, a translate link). The old one is code.
  • Percent-encoded nested URLs (?u=https%3A%2F%2F...) are not judged, so most Google/Facebook/Bing redirect links still pass; only literal nested URLs are refused.
  • Also refused although the client links nothing there: a URL in the title of a masked link ([x](https://allowed.example "https://other.example")), www. tokens (unchanged), code spans and blocks (unchanged), a URL whose port has a sign or whose host is empty (harmless).
  • Nothing in the list was refused for parentheses, punctuation, emoji, IDN, non-ASCII paths, @ in a path (medium.com/@user), Markdown around the link, or angle brackets. Opaque (more than 16 ends) was never hit by an ordinary link: it needs 17 closers inside one authority.

Is judging nested URLs needed for the property? Only where the client can link the inner URL: when the outer link is not linked (a reading rejected or not exact, user info or a hidden character in the current bundle, an outer www. token, a word character before the outer scheme), or when a place the client can end the outer link (a Markdown closer, ||, a framed )) lies between the two schemes. Otherwise the inner URL is part of the outer href and the client never opens it on its own. A narrower rule that keeps the property: judge a nested scheme only in those cases. It is more code in the hottest and riskiest spot; the current rule is safe and documented (doc/moderation.md:345 now says so explicitly).

  • Must change before merge: nothing on over-blocking.
  • Owner's decision: keep "every nested http(s) URL is judged" (about 3% of ordinary links refused under an allow list, about 6% under a block list in my sample, all redirect-style), or take the narrower rule above. For a block list, refusing archive.ph/https://<blocked> may even be wanted (it shows the blocked page).

The "opaque" and "unreachable host" rules

  • Opaque (linkends.go maxEnds): only ever adds a refusal in every mode; truncating the ends list sets the flag, so it cannot drop a reading silently. Not a bypass. A denial of ordinary messages needs 17 closers inside one authority; none of my realistic links came near. OK.
  • Unreachable host ignored by an allow list (compile.go:136, helper reachable): on its own it is sound (I checked in Chromium 158 that hosts with !, ' etc. in the last label stay as written and cannot resolve). But it turns every end mismatch between server and client into an allow-list bypass whenever the server's longer reading ends in such a character, where before the fix such a reading counted against the message. B1 is exactly that. Once B1 is fixed, keep the rule, but it raises the stakes of any future end mismatch; the generator should include long and framed destinations so the sweep can see the next one.

renderMarkdown, current and older bundle, hosts by new URL; Chromium 158 headless where noted)

familyinputsresult
scheme caseHTTP://, hTtPs://judged, refused
scheme split by characterstab inside https, NUL inside httpstab: client links nothing; NUL: both drop it, judged
protocol-relative, other schemes//host, x, ftp:, tel:, mailto:client links only mailto (not a host; not judged, as documented)
dead schemesjavascript:, data: bare, masked, autolinkclient links nothing
non-literal linkswww., bare domain, @name, <#id>, :emoji:, masked titleclient links none; server judges www. and the title (over-block only)
link previewsLinkPreviewCard href = posted URL (server sets p.URL to the posted URL)follows the judged href; author_url comes from the fetched page (not message content)
Markdown structurenested emphasis, escaped closers, \. in host, entities (not decoded by the renderer), code span and block, quote, list, heading, table text, newline in x, [x]\n(url), reference style (not supported), angle brackets, title in parentheses and quotes, nested labels, parentheses in hostall judged; the only miss is B1
tab and newline removalht\ttps://, https://allowed.exa\nmple@other.examplethe renderer never puts tab or newline in an href; no shift
leading/trailing C0 and spacehttps://..., https://\u0001hostno shift
backslash, missing slasheshttps:\\host, https:/host, https:hostclient links nothing
encoded credentials%3A, %40 in user info and hostclient rejects; server refuses (allow)
ideographic and full-width dotsU+3002, U+FF0E, U+FF61, and percent-encoded U+3002same host on both sides
percent-encoded host%6f%74..., %2e, othe%72same host
trailing dotsone and twoserver strips, judged as the listed domain
IPv4 formshex, octal, one, two and three partssame address on both sides
IPv6mapped ::ffff:1.2.3.4, zone id, [::1]zone refused; mapped is printed [::ffff:127.0.0.1] by Go and [::ffff:7f00:1] by the browser (NOTE N2)
empty host, portsempty, :0, leading zeros, +80, -1, 99999as the URL Standard
punycode caseXN--, xN--, invalid xn--zzzsame host
long labels64-character labelsame host (unresolvable anyway)
bidi, zero widthU+200E, U+202E, U+200B, ZWJ, ZWNJclient refuses or browser rejects; server refuses under allow list
ignored code pointsU+034F, U+FE0F, U+180B, U+00AD raw and percent-encodedsame host on both sides (Node and Chromium)
emoji, sharp s (U+00DF), capital sharp semoji label, a label with U+00DF, U+1E9Esame punycode in Go (Unicode 17 tables), Node and Chromium
destination capmasked destination of 2000 unitsB1

Own fuzz: 60,000 messages from my own fragment set (brackets, escapes, quote and list prefixes, fences, entities, ignored characters, ports, user info, masked-link pieces): no message whose client host is a resolvable host outside the rule passed the allow list, block list or block-all, in either bundle.

Tests

  • Flipped tests run as normal tests (go test -tags=integration, vitest). With the fix: TestAnAllowListJudgesTheHostTheClientOpens, TestABlockListJudgesTheHostTheClientOpens, TestBlockAllAndNewAccountsSeeEveryLinkTheClientShows pass (5.2 s).
  • Reverted (compile.go, rule.go, link.ts back to adc32bacc~1, linkscan/linkends/linkhost.go and their unit test and benchmark removed, test files kept): all three fail, 64 failing cases ("let ... through (201)").
  • internal/automod unit tests and the whole markdown vitest directory pass (381 tests).
  • Deep sweep on seeds the author did not use: seeds 201..210, 1,000,000 messages, older bundle: 817,651 with a link, no bypass; seeds 251..260, 500,000 messages, current bundle: no bypass. 1M messages take about 17 s.
  • Nothing weakened: no allowlist entry, no limit raised, no test skipped; the known-failure list is gone because it is empty. gofmt, go vet (with and without -tags=integration), make complexity-check, internal/repo tests, ESLint on the markdown directory, vue-tsc (app and tests), check-english-only: all clean. New files are under the size limits (linkhost.go 220 lines). Commit type fix, English. doc/moderation.md:345 describes the new behaviour, including nested links and the unreachable-host rule.

Actions pinned to full SHAs (the same as ci.yml), permissions: contents: read, persist-credentials: false, no untrusted input in run steps, no secrets, schedule plus manual dispatch, timeout-minutes: 60, 4 shards with fail-fast: false, setup-go from go.mod and Node 26. Run time: about 5 million messages per shard at about 17 s per million, plus setup: a few minutes per shard, well inside the timeout. NOTEs: the seeds are fixed (seeds 1..40 every night), so the job only re-checks the same 20M messages and never explores new ones (consider a date-derived seed offset printed in the log); the workflow is still named "Media group sweep"; vitest does not print the summary line of a passing run (console.log in linkParity.sweep.test.ts:61 is swallowed), so a green run does not show how many messages it checked.

NOTEs

  • N1. Go's IDNA tables are Unicode 17 (go 1.27, x/net v0.62.0), browsers' ICU is older; a code point new in 17 is valid for Go and rejected by the browser (no link: harmless). A mapping changed between versions would make an exact but different host; none found, not exhaustively checked.
  • N2. An IPv4-mapped IPv6 literal is not compared with an IPv4 entry; cleanDomain only accepts names that look like domains, so this matters only if an IP-like name is listed. Unchanged by the fix in effect.
  • N3. Chromium (158) rejects hosts with ZWJ/ZWNJ (CheckJoiners on), as the server assumes; Firefox and WebKit not run.
  • N4. The commit message has a stray line-wrap ("which may stay: the deep run of this fix found that case). More than 16 places to end one authority ...").
  • N5. cleanDomain now uses the new host reader: a configured domain with an invalid port is now refused at configuration time instead of silently cut; ordinary domains, IDN and punycode entries clean as before.

What I ran

All with nice -n 10, go test -p 2, one heavy job at a time, in temporary detached worktrees (removed): the three integration tests with and without the production change; a temporary integration test posting the B1 message under both list modes; temporary Go and vitest probe harnesses (about 260 hand-written inputs, the 71-link realistic set under two allow lists and one block list, the 60,000-message own fuzz) against both bundles; Node 24 and headless Chromium 158 URL parsing of the B1 hrefs and of 16 host edge cases; benchmarks (BenchmarkMatchesLinks plus 18 own worst cases, also against the pre-fix code); the deep sweep (1.5M messages, seeds 201..210 and 251..260); the gates listed above.

What I did not check

Firefox and WebKit (only Chromium and Node); the Windows and Linux desktop shells (same bundle); the DM path and edits through the route (same Engine.Check); the nightly job on GitHub (not run, only read); every code point of the IDNA tables across versions.

Second round: commit 26de20f32

Reviewed: the diff of 26de20f32 only (on adc32bacc): linkscan.go, linkends.go, the new linkcover.go, compile.go, engine.go, linkAuthority.ts, the new linkParity.current.test.ts, the new tests, doc/moderation.md.

Verdict: MERGE. B1, S1, S2 and S3 are closed. Two NOTEs remain, and the over-blocking of redirect-style links stays the owner's decision as documented.

B1 (destination cap): closed

  • Through the real message route, ordinary member, temporary integration test: my round-1 message (masked form, current bundle) and the user-info form for older bundles, under allow_list allowed.example and block_list other.example: 400 in all four. I also tried the angle-bracket framing x, and a variant one unit shorter whose cut falls in the title: 400 in both modes.
  • The cut is counted from the same start as the client's count in every framing. The client's destination must begin with http(s):// (SAFE_HREF is anchored, and a leading space ends the destination, so nothing before the scheme is counted). < is skipped on both sides. The server counts from the scheme to the end of its own authority, which is never earlier than the client's, because the server's stop set holds every character the client's destination stops at. Where the server stops earlier (", <, a control character), the client's cut URL holds that character and SAFE_HREF refuses it. A title is counted as part of the server's authority, which makes the check stricter, not looser.
  • "Authority longer than the cap" is the right condition. If the authority (scheme included) is at most 2000 units, the cut falls at or behind its end and the host is unchanged. Percent escapes are counted as written on both sides; the client decodes them only after the cut. Surrogate pairs are counted as two units (utf16Len), and a cut through a pair is refused (probe: refused in both modes). NUL is dropped first on both sides.
  • The other client caps cannot move a link end to a place the server does not judge:
    • Label limit 500: a longer label is not a link, and its contents are parsed as bare links, which are judged.
    • Inline depth 6: past it, delimiters stay literal, which gives fewer client ends, and the server's readings are a superset.
    • 400-character title window: if the title is longer, the masked link fails and the destination is parsed as a bare link, which the server's framed ends already cover (trimClient(bare)).
    • 6000-character render cap: unreachable within the 4000-character message limit, except with astral characters, where the client then links nothing (over-blocking only).
    • Probes for all four: refused where the client links other.example, in both bundles.
  • My round-1 probe sets rerun on this commit, with only resolvable hosts counted: no bypass in any mode (block-all, allow list, block list) in either bundle. That covers 110 bypass-family inputs, 60,000 own-fuzz messages and 21 new cap probes.

S1 (cost and work budget): closed

My round-1 worst cases, now (links(), -benchtime 100x, against the earlier measurement):

inputbeforenow
3960 non-ASCII letters, 15 closers1.59 ms0.10 ms
280 nested schemes, shared tail1.07 ms0.09 ms
40 nested non-ASCII hosts, 15 closers each0.97 ms0.13 ms

The slowest of all my shapes is now 0.44 ms, for full-width text that only folding turns into a link. That case is unchanged from the code before the fix and lies outside the budget. Links are read once per check; the new-account rule no longer reads them a second time.

Can the budget refuse ordinary messages? Work used by realistic heavy messages (budget 32,768):

messagelinksused
40 mixed ordinary links (Wikipedia, GitHub, YouTube, masked, bold, spoiler, IDN)405%
list of URLs, 3900 characters788%
pasted server log with URLs, 3985 characters687%
Wikipedia links with parentheses, 3942 characters738%
Markdown-heavy (bold masked links, underline, strike), 3916 characters13213%
140 www. domains14018%
redirect-style links, 3960 characters14413%
Cyrillic and Japanese text with 130 IDN links13046%
180 IDN links in bold or italics18265%
Chinese text with 130 links glued to the text (no spaces)130101%: opaque

Only the last one runs out. There, each authority runs on through the following Chinese text up to the next "//" and is charged four times its bytes. That is 130 links in one message, which reads as spam, not an ordinary message. No ordinary shape came above two thirds of the budget.

S2 and S3: closed

  • S2: linkParity.current.test.ts renders the vector file with this bundle on every push. It passes, and the markdown directory passes (390 tests).
  • S3: linkAuthority.ts now checks the decoded authority and the parsed hostname as well. In the current bundle, percent-escaped zero-width spaces, soft hyphens and encoded user info are no longer linked (probes 1-4). Older bundles still link them and the server refuses them.

NOTEs

  • N6. The comment at internal/automod/linkscan.go:20-24 says a nested URL is judged "unless the client surely opens the outer link through it (linkcover.go)". That narrower rule was removed, and linkcover.go holds no such thing: every nested URL is judged, as the commit message and doc/moderation.md say. The comment should say that too.
  • N7. A long message in a script without spaces, where a host-only URL is followed directly by more than 2000 characters of text with no / ? # (Japanese and Chinese write this way), is now opaque and refused by a block list too. Before this commit it passed a block list. The client links a host that includes the text, which leads nowhere either way. This is rare, it is documented as a link "longer than 2,000 characters", and it is the owner's decision. Also not done: the generator still has no long destination (my round-1 suggestion), so the nightly sweep cannot see a future cap regression. The unit test TestAnAuthorityLongerThanTheClientsTargetCapLeadsAnywhere covers this cap.

What I ran (second round)

All with nice -n 10, in a temporary detached worktree (since removed):

  • the B1 reproduction through the real route: 4 message shapes × 2 modes;
  • probe harnesses in both bundles: round-1 sets plus 21 cap probes;
  • the budget measurement above;
  • the benchmarks (BenchmarkLinksWorstCase, BenchmarkMatchesLinks and my 20 shapes);
  • the deep sweep on seeds neither author used: shard 70/100 (seeds 701..710), 1,000,000 messages as older bundles render, 817,479 with a link, no bypass; shard 75/100 (seeds 751..760), 1,000,000 messages as this bundle renders, 817,176 with a link, no bypass;
  • the flipped integration tests and the automod integration tests: pass;
  • gofmt, go vet (with and without -tags=integration), make complexity-check, internal/repo tests, check-english-only, ESLint, vue-tsc (app and tests): clean. New files are under the limits (linkscan.go 211 lines, linkcover.go 70).

Not checked in the second round: Firefox and WebKit, the desktop shells, the nightly job on GitHub.

Something wrong on this page? Tell us.