H-L3-05

From mnema-talk v3.3.13, rendered on this site.

Fix review: H-L3-05 (muted channel follows the resolved mentions) and H-L3-06 (snippets cut after stripping)

Branches: msch128/fix-h-l3-05 (commit 59aeb545b) and msch128/fix-h-l3-06 (commit 7a91dd9fa), both on top of 3be5cb096 (the lane 3 test state rebased onto main, not reviewed here). Only the two fix commits were reviewed. Read-only review in two detached temp worktrees (npm ci in web/ in each), removed afterwards. Probe tests were written into the worktrees only and deleted with them. Nothing committed or pushed.

FixVerdict
H-L3-05MERGE AFTER FIXES: B1 (test typecheck red). The fix itself is right.
H-L3-06MERGE AFTER FIXES: B2 (the server's parser now decides what every web client shows in a snippet, and it disagrees with the web renderer), S2, S3

1. Does it remove the cause

Cause (finding, refuter): messageMentionsMeDirectly matched @username in the raw text and never looked at msg.mentions, the list the server resolved under the sender's mention policy. The fix adds one guard (web/src/stores/chat/notifications.ts:42): when the payload carries a mentions array without me, the message is not addressed to me. A reply to me still counts first. Being on the list is necessary, not sufficient; the old text and role checks then separate a name or role mention from @all/@here.

This is the right place: the same function feeds every variant the refuter listed: the muted badge (events.ts:146), the page's OS notification (notifications.ts:76, shouldNotify lets mute through only on isDirectMention), the sound (messageSound.ts:36), the thread-follow filter (threadAllows) and suppress_everyone (messageMentionsMe delegates to it). Reasoning against the server:

  • The wire list is authoritative on every live path. Every message_create broadcast is a message loaded back from the database (GetMessage in post.go publishCreated/PublishMessage, webhook/execute.go:227, bot/interactions_respond.go:191, poll/service.go:77, outbox), and loadMentions fills Mentions from message_mentions (messages.go:135-148, initialised to []). The poll's empty-list fallback is only the author's own HTTP response, not broadcast.
  • Server rule for a mute (readstate.go:47-49, push/recipients.go:177): addressed = a reply to me, or a message_mentions row of kind name (0) or role (2). The server stores kind 0 only for listed users whose name is in the text (queryMentioned CASE) and kind 2 for listed role members (addRoleMentions). So "listed AND (text names me OR a mentioned role is mine)" is the server rule, with the exception in N1.
  • Role mention over the recipient cap: capRecipients drops users but keeps Roles, so mention_roles still lists my role while I am not in mentions. Before the fix the client counted it, the server did not; after the fix both do not. Same for the 100-name cap (maxMentionNames) and for CanView.
  • Older server without the list: Array.isArray is false, the text rule stays (probe P3).

Own probes (not in the commit's tests; scratch test in the worktree, deleted):

ProbeWith fixFix reverted
P1 muted, @Dev ping, mention_roles: [myRole], mentions: [] (role dropped by the cap)badge 0, no notificationfails (badge 1)
P2 muted, same with mentions: [me]badge 1passes
P3 muted, hey @max, no mentions field (older server)badge 1passes
P4 muted, @all @Dev deploy, mention_roles: [myRole], mentions: [me]badge 1 (see N1)passes
P5 suppress_everyone on, level mentions, webhook build failed, @max, mentions: []badge 0, no notificationfails
P6 level mentions, same message (control, unchanged path)badge 0passes
P7 followed-threads known, reply in an unfollowed thread with unresolved @maxno notificationfails

So the cause is removed for every variant listed, including role mentions, thread follows and suppress_everyone, which the commit's tests do not cover.

2. Recorded tests

web/src/stores/chatMentionPolicy.test.ts: the two it.fails are plain it now; three cases added (live badge equals the badge after a reload, @all alone keeps the mute, a reply to me breaks it). With the production change reverted (notifications.ts from 3be5cb096): 3 failed, 13 passed (the two H-L3-05 cases and the reload case fail, as the commit message says). With the fix: the five files around it (chatMentionPolicy, chatThreadFollow, chatComfort, chatLogic, messageSound) 68 passed.

chatThreadFollow.test.ts fixture (lead's question): the reply hello @me came from an ordinary member (fixtureId(9)) to the signed-in user whose username is me, with mentions: [] taken from messageFixture's default. The server never sends that: an ordinary member's @me resolves (active user, can view the channel; the refuter showed the server's parser is not Markdown aware), so the list holds me. The new fixture (mentions: [me]) is the server's real payload. It is not a weakened test: the case still checks that a name breaks through the thread filter, and P7 checks that an unresolved name does not.

3. Does it introduce a defect

No behaviour change beyond the finding. No API change (no wire field). messageMentionsMe (unmuted path) is unchanged. Cost: one Array.includes per message.

B1 (BLOCKER): the new test does not typecheck

web/src/stores/chatMentionPolicy.test.ts:97 builds reply_to: { id, user_id, username, display_name, content } without deleted, which ChatReplyPreview requires (internal/chat/core/message_types.go:166, json:"deleted"). npx vue-tsc --noEmit -p tsconfig.tests.json fails with TS2322 at 97:50 (1 error; the app config passes). This config is part of npm run typecheck, so CI is red. Fix: add deleted: false.

4. Gates

ESLint on the three touched web files: clean. vue-tsc app config: clean; tests config: B1. English check: pass. notifications.ts 124 lines, test 100 lines. Commit type fix. doc/architecture.md gains a paragraph that matches the new behaviour.

5. The residual the author reports ("@all @Role"), and other mismatches

N1: "@all @Role" is a real server defect, the documents support the client; worth its own finding

addRoleMentions (internal/chat/mentions/mentions_roles.go, the have[id] skip) leaves a role member that @all/@here already put in the set at kind group (1). The muted badge (readstate.go:49, kind <> group) and the push (namedMentions, push/recipients.go:177) then do not count them, so @all @Dev keeps the mute of a muted member of Dev, while @Dev alone breaks it. doc/architecture.md:594-597 ("addressed to the member (by name, through a role they hold, or as a reply)") and doc/upgrade.md:1041-1045 say the member is addressed through the role. The client (before and after this fix, probe P4) counts it, so live badge and reload disagree in the other direction. Code-read, not run. Suggested severity S4 (needs a sender who may mention everyone and both mentions in one message; effect: a missed notification, plus a live/reload disagreement). Server fix direction: when a role member is already in the set at kind group, raise the kind to role instead of skipping. Not this fix's job; the commit message states it honestly.

N2 (candidate finding): webhook/bot with user mentions off still addresses a named member when the message also says @all

mentionSkipped (internal/chat/mentions/mentions.go) does not skip a user that @all reaches, and queryMentioned gives kind 0 (name) to every listed user whose name is in the text, whatever the policy. So from a webhook with allow_user_mentions off but allow_everyone on (or a bot with allowed_mentions.parse: ["everyone"] only), @all @max stores max as a name mention and breaks max's mute (badge after reload, push, and the client agrees). doc/webhooks.md:34: off "turns every member mention into plain text". Code-read, not run. S4 (unusual configuration). Pre-existing, not caused or changed by this fix.

N3: 100-name cap with @all

A message with @all, 100 other names and then @max: the server reads only 100 names (capNames), so max is listed through @all (kind group) and not counted; the client sees @max in the text and counts it. Live badge 1, reload 0. Needs MentionEveryone and 101+ names; noted, not worth a finding.

6. Not checked (H-L3-05)

Server push and read-state queries were read, not run. The desktop client (it loads the web bundle). No end-to-end run with a real webhook.


H-L3-06

1. Does it remove the cause

Cause: raw Markdown cut to 200 characters before stripping. The fix adds mdplain.Snippet (Go, internal/mdplain/snippet.go) and markdownSnippet (web, web/src/shared/lib/markdown/snippet.ts): a message of at most 200 code points is returned as it is; a longer one is reduced to plain text (spoilers as a private-use mark), white space collapsed, cut to 199 plus "…", then escaped back into Markdown with each spoiler written as ||…||. All sites named by the finding and the refuter use it: reply quotes (messages_scan.go:35; MessageSelect reads rm.content), the client's edit path (replies.ts markPreviewEdited), search roots (search_roots.go:53), reminders (later/visibility.go:55, which feeds the list, the in-app notice and the reminder push), plus the conversation list (dm.go:202 and the client's lastMessageOf), which the finding missed. Saved items, forwards, message pushes and OS notifications already stripped the whole message (refuter's table), confirmed.

Against its own parser on each side the fix holds:

  • Go deep sweep, run myself: PLAIN_SWEEP=1 go test -run TestSnippetNeverShows ./internal/mdplain -> "100000 messages, 13519 cut, 0 leaks". Round-trip test (20,000 strings) passes.
  • Web deep sweep, run myself: PLAIN_SWEEP=1 PLAIN_SWEEP_N=100000 npx vitest run plainSpoiler.test.ts snippet.test.ts replies.test.ts src/dm: 8 files, 126 tests passed (the sweep asserts zero leaks; the reporter does not print its console line for a passing test).
  • My own web generator (a richer alphabet: + , 2) , -#, entities, <@id>, <:emoji:id>, <t:…>, autolinks, ZWJ families, flags, combining marks, surrogate pairs, fences, escapes; 100,000 messages, 85,450 cut): 0 snippets whose previewText shows a hidden canary, 0 whose renderMarkdown contains one, and previewText(snippet) equals the expected cut plain text in every case.

But the server's snippet is built by the Go parser, and every client now shows what the Go parser decided. That is B2.

B2 (BLOCKER): the server's parser now decides what web clients show, and it disagrees with the web renderer

Before the fix, the web client stripped the (raw) reply quote with the web parser, the same parser that renders and hides the spoiler. After the fix, for a message over 200 characters, the server reduces it to plain text with mdplain.Plain and the client only reads that back. The commit's premise ("any reader that strips it with this flavour ... gets exactly the cut plain text") holds only if mdplain.Plain hides exactly what the web renderer hides. It does not. Each side's sweep checks the side against itself, so neither can see this.

Cross-check (my generator, 100,000 messages; oracle: the web previewText of the whole message, which the commit's own test ties to the renderer):

  • the Go snippet shows text the web renderer hides in 940 messages;
  • the web reply preview leaked before the fix in 3,267 messages, after the fix in 940: 3,061 fixed, but 734 messages that were hidden before now leak;
  • mdplain.Plain of the whole message (the message push, pre-existing) disagrees in 978.
  • Go and web snippets differ for 2,199 of the 100,000 (so a reply quote also changes text when the client rebuilds it after an edit).

One class, minimised: a precedence difference between emphasis and code spans when the emphasis spans a paragraph break. The web parser lets the code span win, mdplain the emphasis, so the || inside the code span pair up differently and text between spoiler markers comes out visible. Minimal input and its outputs on both sides are in a separate, non-public note for the coordinator (this fix is not released). Not all 734 are classified; other classes may exist.

Concrete failing scenario: a member writes a long message (over 200 characters) of that shape with a spoiler inside the first 200 characters. Before the fix every reader's reply quote hides the spoiler; after it, every web and desktop client shows its text in every reply quote, the conversation list, the search root and the reminder list. Same S3 effect as the finding, new trigger.

Fix direction: make mdplain.Plain agree with the web parser (here: code spans before emphasis), and add a cross-implementation check, for example a generated corpus checked into testdata with the web renderer's hidden set (or the web previewText) recorded, checked by a Go test against Plain and Snippet. The shared snippet.json has 14 hand-written vectors and does not catch it. This also fixes the pre-existing push leak, which deserves its own finding (see N6).

2. Recorded tests

Go internal/later/snippet_spoiler_test.go (inverse assertion) became a plain check on mdplain.Snippet; web plainSpoiler.test.ts it.fails became plain it; new snippet_test.go, snippet.test.ts, testdata/snippet.json, and an integration test TestAReplyQuoteHidesASpoilerThatClosesAfterTheCut.

Revert check, done: Snippet and markdownSnippet replaced by the raw cut (textcut.Runes(content, limit-1)+"…" / clampChars). Go: TestSnippetAgreesWithTheSharedFixtures, TestSnippetNeverShows... and internal/later TestASnippetHidesASpoilerThatClosesAfterTheCut fail. Web: 16 tests fail (plainSpoiler's two cases and the fixture vectors). Restored afterwards.

Run with the fix: go test -p 2 ./internal/mdplain ./internal/later ./internal/chat/messages ./internal/chat/search ./internal/chat/dm ./internal/push ok; with -tags=integration (OrbStack): ./internal/later ./internal/chat/dm ./internal/chat/search ./internal/chat/messages ok, and ./internal/server -run 'Reply|Search|DM|Dm|Later|Remind|Snippet|Quote' ok, including the new integration test.

3. Does it introduce a defect (other than B2)

API (lead's question)

doc/api-stability.md: field types are unchanged, so the change is not breaking by its rules; it is a change of what a documented string contains, which the commit states in doc/upgrade.md and the OpenAPI descriptions (make openapi-check passes; npm run contracts:check passes; event golden tests pass).

  • Old web clients and the desktop client (which runs a web bundle): I copied the strip code of v1.0.0 and v3.0.0 (web/src/lib/markdown*, replies.ts) and ran their previewText on 85,450 new snippets: identical to the new client's output in every case. No stray backslashes: every character the escaper prefixes is a symbol both old parsers already treat as escapable (any non-alphanumeric, non-space character), and ||…|| reads as a spoiler there.
  • Bots, outgoing webhooks, third-party clients that render these fields as Markdown: see S3.

S2 (SHOULD-FIX): worst-case cost on read paths is about 80 times the measured figure

Benchmark confirmed: BenchmarkSnippetOfALongMessage 27.5-28.5 µs/op, 36.7 KB, 261 allocs (M-series Mac, 3 runs). But mdplain.Plain is quadratic on some inputs, and the four queries now run it on up to 4,000 characters per row, per reader, per load (before: 200). Measured Snippet on 4,000-character inputs: [a]( repeated 1.6 ms, [ repeated 0.86 ms, mixed openers 0.1 ms. Plain on [a]( repeated: 500 chars 44 µs, 1,000 176 µs, 2,000 707 µs, 4,000 2,260 µs (quadratic). Anyone who can post can make a message page that quotes 50 such messages cost about 100 ms of server CPU for every reader who opens it, and the same on the DM list and search roots. Before the fix this cost was paid once per message by the push only. Not a blocker (bounded by MaxMessageLen and page size), but the commit's "1.3 ms per page" is the typical case, not the bound. Options: make the link-opener scan in mdplain linear (it already memoises misses for other openers), or build the snippet once at write time, or at least memoise per quoted message within a query.

S3 (SHOULD-FIX): document that a snippet can exceed 200 characters and is no longer Markdown from the message

  • The escaped snippet is longer than its plain text: up to about 5 characters per plain character (each spoiler mark becomes ||…||, each symbol gets a backslash). Measured: 996 characters for a long run of |, 399 for runs of ``, *, <. The old OpenAPI text said "at most 200 characters" for last_message.text, the search root snippet and the reminder snippet. api-stability.md counts a larger maximum as not breaking, but doc/upgrade.md should say it, and the new descriptions ("the first 200 characters of its plain text as Markdown") can be read as a 200 bound.
  • A reader that renders the snippet as Markdown (a bot, a third-party client; not ours: every web site strips it) now sees text the original showed as code: a URL that was inside a code span becomes a live autolink (`https://x.example` gives a bare link; the link text is the URL itself, so no disguise), and an @everyone inside code becomes a mention pill (display only: the server's notification is decided on the message, and its parser already treats code-span mentions as real). No hidden text, no new notification, and custom emoji, <@id> and <t:…> tokens are neutralised by the escaped <. doc/upgrade.md names bots; it should also name outgoing webhooks (message_create deliveries carry reply_to.content).

Escaping, edges (lead's questions)

  • Can a crafted message make the snippet contain hidden text, a link or a notifying pill: against the web parser, no (B2 aside, which comes from the Go parser, not the escaper); links and pills only as in S3. A spoiler in the snippet holds only "…": clicking it reveals "…".
  • 200 and 201 characters: 200 code points is returned raw (spoiler intact, hidden by the reader); 201 goes through plain, ends … ||…|| and reads "Spoiler". Checked in Go.
  • Surrogate pairs, ZWJ sequences, flags, combining marks at the cut: both textcut.Runes and clampChars back up to a character boundary; checked with emoji, a ZWJ family, a flag pair and e + U+0301 at the cut: no broken character.
  • &#57344; (an entity for the private-use mark) stays literal in both parsers; a U+E000 in the message is dropped before marking (test exists).
  • A long message whose plain text is empty (for example a long [) gives an empty snippet; with omitempty the reply quote then has no content. Harmless.

4. Gates

gofmt clean; go vet with and without -tags=integration clean on touched packages; make complexity-check exit 0; internal/repo tests pass; ESLint clean on the seven touched web files; vue-tsc app and tests configs clean; English check pass. New files: snippet.go 69 lines, snippet.ts 49. Commit type fix. No allowlist entry, no raised limit, no skipped test.

5. The two never-stripped sites (lead's question)

N4: event comment delete dialog (candidate finding, S4)

web/src/events/components/EventComments.vue:62 puts c.content.slice(0, 160) raw into the confirm dialog, while the comment itself renders through MarkdownContent (spoilers hidden). Whoever deletes a comment (its author or a moderator) sees the spoiler text and the Markdown source. The forum's equivalent (useForumPostActions.ts:67) uses previewText(...). Also slice cuts UTF-16 units and can split a surrogate pair. Small, real, outside H-L3-06's mechanism; worth an S4 finding with a one-line fix (previewText, then a character-aware cut).

N5: wiki search snippets (harmless or S4)

internal/wiki/store_search.go:48-57 returns a raw 160-rune window around the first match and WikiSearch.vue:52 shows it as text, so a ||spoiler|| near the match shows its text and the Markdown markers. Everyone who gets the hit can read the page. Arguably a raw search excerpt, not a plain form. I would file it together with N4 as one S4 "raw excerpts show spoilers" finding, or leave it.

N6: pre-existing push leak (candidate finding, S3)

The B2 disagreement already reaches the message push today: mdplain.Plain of the whole message shows text the web renderer hides in 978 of my 100,000 generated messages (push/payload.go:222). Not caused by this fix, but the same fix closes it. Suggest a separate finding so the cross-implementation property gets its own test.

6. Not checked (H-L3-06)

End-to-end push delivery; the native parts of the desktop client (only its web bundle's strip code, by copying v1.0.0 and v3.0.0); whether every insert path enforces MaxMessageLen (the cost bound in S2 assumes it); a classification of all 734 regressions beyond the one minimised class; the web markdownSnippet cost on pathological input (it runs only on the client's edit and conversation-list paths).


Summary

  • H-L3-05: MERGE AFTER FIXES. B1 test typecheck red (deleted missing at chatMentionPolicy.test.ts:97). N1 "@all @Role" real server under-count, documents back the client, own finding (S4). N2 user mentions off plus @all still addresses by name (candidate S4). N3 100-name cap mismatch (note).
  • H-L3-06: MERGE AFTER FIXES. B2 Go parser now decides what web clients show and disagrees with the web renderer: 734 of 100,000 generated messages newly leak (3,061 fixed). S2 quadratic worst case on read paths (about 2 ms per snippet). S3 document longer snippets and Markdown rendering by bots and webhooks. N4 event comment delete dialog shows spoilers (candidate S4). N5 wiki search raw window. N6 pre-existing push leak from the same disagreement (candidate S3).

Second round

New commits reviewed: 423118a9a (fix-h-l3-05), 5a763ac81 and 70b7dc072 (fix-h-l3-06). Fetched first; fresh detached worktrees with npm ci, removed afterwards; probe files and generated corpora deleted. Nothing committed or pushed.

FixVerdict
H-L3-05MERGE (B1 closed; the coordinator folds the test-only fix: commit)
H-L3-06MERGE (B2, S2, S3 closed; NOTEs N7 to N9 only)

H-L3-05: B1 closed

423118a9a adds deleted: false to the reply preview at chatMentionPolicy.test.ts:97; nothing else changes. vue-tsc --noEmit with tsconfig.app.json: exit 0; with tsconfig.tests.json: exit 0 (it failed before with TS2322). chatMentionPolicy and chatThreadFollow tests: 16 passed. The commit type is fix: for a change that only touches a test; it should be test:. As agreed, the coordinator folds it into 59aeb545b.

H-L3-06: B2 closed

5a763ac81 replaces the server's mdplain with a port of the client's parser (blocks, inline, link, plain, working on UTF-16 units). Where only a browser can judge a link host and that host could move a spoiler, or where the work bound runs out, it fails closed: everything between the first and the last || counts as hidden. Both are explained in plain.go's package comment.

My own cross check, on seeds neither of us used

This is the same generator and oracle that found the 734 regressions in the first round: the web previewText of the whole message, plus the renderer-HTML verdict from the new fixture, whichever hides more. Added this time: visible canaries in plain text, so that dropped visible text can be measured. Seeds 301, 302 and 303 (100,000 messages each, about 85 % longer than 200 characters), plus 100,000 messages without any | (seed 304). Main's server side is a copy of the old mdplain (7a91dd9fa). Main's reply preview is the web previewText of the first 200 code points.

CorpusNew push leaksNew snippet leaksMain push leaksMain reply-preview leaks7a91dd9fa snippet leaksRegressions (push / snippet)
seed 301001,33112,5791,2870 / 0
seed 302001,31012,6601,2660 / 0
seed 303001,30912,6421,2630 / 0
seed 304, no ``00000

The two oracles never disagreed in either direction.

The oracle cannot pass by hiding everything

In my corpora, the new mdplain.Plain equals the web previewText character for character (after collapsing white space) in all 400,000 messages. The Go snippet's plain form equals the web snippet's in all 400,000. Visible canaries the web shows: 1,519,026. Visible canaries the new form drops: 0. Characters lost: 0 of 86.8 million.

Main's old mdplain, by contrast:

  • dropped visible text in about 1.9 % of the messages with spoilers (1,831 to 1,898 per 100,000);
  • differed from the web in 5,744 of the 100,000 messages without |.

So the claim that "messages without || are unaffected" holds in the sense the lead asked about: nothing in them is blanked or hidden. Their push text does change for about 6 % of my generated messages without |. It changes toward the web renderer (N7).

My generator has no hosts that only a browser can judge, so nothing failed closed there. To look at the fail-closed share, I ran the author's generator (plainCross.fixture.ts) on new seeds 7101 to 7103 (100,000 messages each):

  • the author's TestPlainHidesWhatTheRendererHides gives leaks 0, failClosed 908 and 870, plainDiffers 0, snippetDiffers 0;
  • my analysis of the same records gives leaks 0 and regressions 0; main's push leaked in 9,794 to 9,960 messages.
  • About 0.9 % of messages differ from the web (898 to 921), and nearly all of them contain ||. That is the fail-closed share.

What a reader sees in a fail-closed message (N8):

  • the text before the first ||, one "Spoiler", then the text after the last ||;
  • in those messages, the median loss is 52 to 54 % of the visible text (p90 86 to 87 %, max 98 %); the median is about 140 characters, the max about 520;
  • about 0.75 % of all messages lose at least one visible canary.

The triggers are a masked link or an autolink whose host the browser must judge: non-ASCII, punycode, a percent sign in the host, or IPv6. A bare URL fails closed only when the uncertain reading could move markup. For real messages that means a link like that plus two or more spoilers with text between them. A message with a single spoiler loses nothing.

mdplain's callers

grep mdplain. outside the package finds exactly six callers:

  • push/payload.go:222 (plainLine);
  • onboarding/sharepage.go:62 (the public share page's description of the welcome text);
  • messages_scan.go:35, search_roots.go:55, dm.go:204, later/visibility.go:57 (the snippets).

There is no search indexing, mail text or other notification text that uses it. All of them now get the web's reading: the same visible text except in the fail-closed case. The share page also loses nothing visible, because its spoiler word is "…" and fail-closed only affects text between spoilers.

Tests I ran:

  • unit tests: ./internal/mdplain ./internal/push ./internal/onboarding ./internal/later ./internal/chat/... ./internal/repo all ok;
  • integration tests (-tags=integration): ./internal/later ./internal/chat/dm ./internal/chat/search ./internal/chat/messages ./internal/push ./internal/onboarding ok, and ./internal/server with -run 'Reply|Search|DM|Dm|Later|Remind|Snippet|Quote|Push|Share|Onboard' ok.

Golden data and fixtures:

  • plain.json gains 12 vectors. No existing vector changed.
  • snippet.json changes one vector, by the escaped : of S3, and adds one.
  • plain_test.go only renames a helper test (hasPrefixFold to hasAtFold).
  • No push or onboarding expectation was edited.

So no golden data was adjusted to hide a change.

Work bound (S2 closed)

Here is what I measured for Plain and Snippet at 4,000 and 6,000 characters. 6,000 is the parser's own limit; above 6,000 UTF-16 units both sides leave the text unparsed, as before.

  • My first-round worst cases at 4,000: [a]( 116 µs (was 2,260 µs for Plain, 1,635 µs for Snippet), [ 73 µs (was 834), mixed openers 130 µs.
  • Seven new hostile shapes of my own, at most 126 µs at 4,000 and 187 µs at 6,000:
    • backtick runs of growing length;
    • <https:// openers;
    • failing bare URLs;
    • deeply nested masked links;
    • escaped pipes;
    • quote and list markers;
    • IDN masked links between spoilers.
  • Steps never exceeded 30,000 against a budget of 260,096 (4,000) and 388,096 (6,000). Time grows linearly from 4,000 to 6,000.
  • A dense ordinary message: 62 µs (66 µs as a snippet), against 27 µs for the old mdplain. That matches the author's 0.026 to 0.064 ms.

The new TestPlainWorkIsLinearInTheMessage pins twelve more hostile shapes.

S3 closed

The escaper now also escapes the : of http(s)://. In the web renderer, the snippet of a long message contains no <a for each of these: a URL in a code span (lower and upper case), an autolink in a code span, a masked link in a code span, http://, and a message with a real link and a bare URL. Old clients (v1.0.0, v2.9.12, v3.0.0) treat : as escapable (/[^0-9A-Za-z\s]/), so https\:// reads as https:// there. MaxSnippetUnits = 996 is pinned by a test that also requires the all-spoiler case to reach exactly 996. doc/upgrade.md now names outgoing webhooks, states the 996-character bound and explains the fail-closed rule.

Push in both languages

internal/push/payload_hidden_test.go runs the 1,000-record corpus through clean(plainLine(..)) for en and de and fails on any hidden canary. It passes.

Nightly workflow (70b7dc072)

  • Triggers: schedule and workflow_dispatch only.
  • Permissions: contents: read.
  • Checkout uses persist-credentials: false.
  • Actions are pinned to full SHAs, the same SHAs the other 26/8/12 uses in the repository's workflows (checkout v7.0.1, setup-node v7.1.0, setup-go v7.0.0).
  • No secrets and no untrusted input. The seed is 1000 + GITHUB_RUN_NUMBER, expanded by the shell from the environment.
  • timeout-minutes: 60.
  • The npm cache is used as in sim-sweep.yml.
  • Expected run time: locally, writing 100,000 records took 8 s and the Go check 3.4 s, so 200,000 is a few minutes plus npm ci.
  • AGENTS.md lists the workflow.

Gates

  • gofmt clean.
  • go vet with and without -tags=integration clean.
  • make complexity-check exit 0.
  • internal/repo ok.
  • English check passes.
  • make openapi-check not stale.
  • ESLint on the three web files clean.
  • vue-tsc app and tests configs exit 0.
  • Web markdown, replies and dm tests: 452 passed, 4 skipped (env-gated).

Production files are at most 339 lines. The only lines over 150 characters are two first-round lines in snippet_test.go. The split functions read normally: no packed statements and no long one-liners.

NOTEs

  • N7: push texts change for ordinary messages. The parser swap changes the push text of messages without spoilers too. In my generated corpus without |, 5.7 % of messages read differently from main. Every such change brings the text to exactly what the web shows, and nothing visible is lost. Release notes could say "push texts now read like the message view" rather than "unaffected".
  • N8: what fail-closed costs. About 0.9 % of the author's generated messages fail closed, and they lose about half their visible text to one "Spoiler". In real messages this needs a link with a host only a browser can judge, plus two or more spoilers. The cost is acceptable for a rule that cannot leak. It could be narrowed later by failing closed only around the spoiler pairs the uncertain link can reach.
  • N9: 423118a9a commit type. It is fix: for a test-only change. The coordinator folds it.

The first round's candidate findings stand unchanged: N1, N2 and N4 to N6. N6, the pre-existing push leak, is fixed by 5a763ac81 together with B2, so it now needs only a record, not a fix.

Not checked (second round)

  • The 0.9 % fail-closed set is not classified beyond the triggers in the code.
  • No end-to-end push delivery.
  • The nightly workflow was not run on GitHub (only its steps, locally).

Third round

One commit: b01ffa11a on msch128/hunt-harness-1 (parent f0714dd85), fetched first. It ports the client's new authority rule (linkAuthority.ts plainAuthority: no user info, percent escapes that decode as UTF-8, no \p{Cc}/\p{Cf} in the authority as written or decoded) into internal/mdplain/host.go, and makes Go decide hosts that are ASCII once percent-decoded. It also adds the drift guard (hrefParity.test.ts, which writes testdata/hrefs.json, checked by TestSafeHrefAgreesWithTheClient) and extends the cross generator. Fresh detached worktree with npm ci, removed afterwards together with all probes and corpora. Nothing committed or pushed.

Verdict: MERGE (H-L3-06 with this commit; no BLOCKER, no SHOULD-FIX; NOTEs N10 and N11).

My cross check on this state

Same generator, oracle and comparison as the second round. Seeds 401 to 403 (100,000 messages each). Seeds 404 and 405 (100,000 each) are link-rich: 40 % of the pieces are taken from 32 link shapes:

  • user info (u:p@, a.example@b.example, @h, user info in masked links and autolinks);
  • percent-encoded authorities (%2E, %41, %40, %2540, %00, %2F, %3A, %E2%80%8B, %zz, %C3%A9, the overlong %C0%80, encoded IPv4 shorthand, upper-case hex);
  • control and format characters raw in hosts (U+200B, U+00AD, U+2060, U+200E, U+0085, U+0007);
  • IDN hosts.

"Main" means origin/main's mdplain for the push. For the reply preview it means LEFT 200 followed by this branch's web strip.

CorpusNew push leaksNew snippet leaksMain push leaksMain reply leaksRegressions (push / snippet)Oracle disagreements
seed 401001,62612,2700 / 00
seed 402001,65412,3200 / 00
seed 403001,70412,1510 / 00
seed 404, link-rich00626,5610 / 00
seed 405, link-rich00826,4700 / 00

Ordinary corpora (401 to 403):

  • the new plain form and snippet equal the web's character for character in all 300,000 messages;
  • the new form drops 0 of 1,230,045 visible canaries;
  • main dropped visible text in about 2.3 % of these messages.

Link-rich corpora (404, 405):

  • about 16 % of messages differ from the web, nearly all of them by failing closed;
  • 12.4 % lose at least one visible canary, about 7 % of the visible characters in all.

I traced that back. Of the 32 link shapes, exactly one makes Go unsure enough to fail closed: a masked link to a non-ASCII host ([x](https://café.example)). My generator draws it often, and most messages carry several spoilers. None of the percent-encoded, user-info or control/format shapes makes Go unsure any more. Before this commit, any % in a host did. So the fail-closed share depends on how common masked IDN links are, as in N8. The 0.4 % the author measured is plausible for their generator.

In 10 link-rich messages without ||, the Go text differs from the web's because a masked IDN link is shown as source instead of its label. That is the documented "unsure: may show text differently, never what is hidden" case (N10).

Can Go now be sure and wrong?

I recorded the client's safeHref answer for 60,216 targets and compared them with the Go port wherever it says it is sure. The targets: 71 hand-written authorities, each under http://, https:// and HTTPS://, plus 20,000 random authorities built from % %2E %2F %40 %3A %25 %00 %41 %C3%A9 %E2%80%8B %7F %5C %5B %5D @ : [ ] \ 0x, U+200B, U+00AD, U+0085 and letters, digits and dots.

The hand-written cases:

  • %2540 and %252E (double encoding);
  • %00;
  • encoded dots, including %2E%2E, a trailing %2E and lower-case %2e;
  • encoded /, \, @, :, #, ?, %, [, ] and |;
  • encoded port separators, and ports with %, +, -, a space and leading zeros;
  • encoded IPv4 shorthand: %31%32%37.1, 0x%37f.1, %30X7F.1, 127.%31, 4294967295 and 4294967296 encoded, and a.%31;
  • encoded DEL, TAB and LF;
  • NBSP, soft hyphen, BOM, ZWSP and emoji percent-encoded;
  • xn%2D%2D;
  • backslash before @;
  • user info, written and encoded.

Result: 0 answers where Go is sure and wrong, and 3,672 left to the browser. The decoding mirrors the URL parser:

  • an encoded @, /, :, %, \ or [ decodes to a forbidden domain code point, so the host is refused, as the browser refuses it;
  • %2540 decodes once to %40 and is refused for its %;
  • %00 is refused by the Cc rule;
  • encoded dots and digits feed the same label and IPv4 logic as written ones;
  • the port is split on the raw : before decoding, as in the URL parser.

Cc and Cf, Go against JavaScript

linkAuthority.ts tests /[\p{Cc}\p{Cf}]/u. Go's hidden tests unicode.Cc and unicode.Cf. I compared the two over every code point:

  • Go 1.27.4 has Unicode 17.0.0, Node 24.12 has Unicode 16.0 (ICU 77.1).
  • Both sets have 235 code points and are identical, with no code point in only one of them.

A user's browser may run an older Unicode version, where a few code points that are Cf now (for example the Egyptian format controls U+13439 to U+1343F, added in 15.0) are unassigned. There Go refuses the link as "hidden" while the old browser's regex does not match. But an unassigned code point is disallowed by that browser's IDNA, so it refuses the host too, and both readings agree that it is no link. In a path or query the rule does not apply on either side. No disagreement follows (N11).

The drift guard

I mutated linkAuthority.ts two ways:

  1. dropped the authority.includes('@') test;
  2. narrowed HIDDEN to \p{Cc}.

Each time hrefParity.test.ts failed ("the recorded answers are the client's"). After HREF_UPDATE=1, Go's TestSafeHrefAgreesWithTheClient failed: 3 or more mismatches after mutation 1, such as safeHref("http://@h.x/") = false, the client says true, and 8 after mutation 2. Restored afterwards. So a change to the client rule cannot pass both sides unnoticed.

Tests and gates

  • Full vitest, 3 workers, once: 662 files passed, 2 skipped; 12,025 tests passed, 8 expected fail, 98 skipped; 1 min 54 s.
  • Go: ./internal/mdplain ./internal/push ./internal/onboarding ./internal/later ./internal/chat/... ./internal/repo ./internal/automod ok.
  • gofmt clean, go vet clean, make complexity-check exit 0, English check passes.
  • ESLint clean on the two web files; vue-tsc tests config exit 0.
  • host.go is 197 lines.
  • cross.jsonl was regenerated with the extended generator. That is not a hidden golden change: the web test requires it to equal what the renderer says, record for record.
  • doc/upgrade.md now names percent-encoded non-ASCII hosts and punycode for the fail-closed rule.

NOTEs

  • N10: masked IDN links read as source. In a message without spoilers, Go may show a masked link to a non-ASCII host as its Markdown source where the web shows the label. This is by design (text may differ, nothing hidden is shown). It affects only push text and snippets.
  • N11: older browsers' Unicode tables. The Cf set of the user's browser may be older than Go's. Code points that are Cf in new versions are unassigned, and so refused by IDNA, in old ones. Both sides then refuse the link.

Not checked (third round)

  • Real browsers other than Node's URL implementation; the guard and my comparison both use Node.
  • IDNA mapping of non-ASCII hosts: Go leaves those to the browser by design.

Something wrong on this page? Tell us.