H-L3-05
From mnema-talk v3.3.13, rendered on this site.
Fix review: H-L3-05 (muted channel follows the resolved mentions) and H-L3-06 (snippets cut after stripping)
Branches: msch128/fix-h-l3-05 (commit 59aeb545b) and msch128/fix-h-l3-06 (commit 7a91dd9fa), both on top of
3be5cb096 (the lane 3 test state rebased onto main, not reviewed here). Only the two fix commits were reviewed.
Read-only review in two detached temp worktrees (npm ci in web/ in each), removed afterwards. Probe tests
were written into the worktrees only and deleted with them. Nothing committed or pushed.
| Fix | Verdict |
|---|---|
| H-L3-05 | MERGE AFTER FIXES: B1 (test typecheck red). The fix itself is right. |
| H-L3-06 | MERGE AFTER FIXES: B2 (the server's parser now decides what every web client shows in a snippet, and it disagrees with the web renderer), S2, S3 |
1. Does it remove the cause
Cause (finding, refuter): messageMentionsMeDirectly matched @username in the raw text and never looked at
msg.mentions, the list the server resolved under the sender's mention policy. The fix adds one guard
(web/src/stores/chat/notifications.ts:42): when the payload carries a mentions array without me, the
message is not addressed to me. A reply to me still counts first. Being on the list is necessary, not
sufficient; the old text and role checks then separate a name or role mention from @all/@here.
This is the right place: the same function feeds every variant the refuter listed: the muted badge
(events.ts:146), the page's OS notification (notifications.ts:76, shouldNotify lets mute through only on
isDirectMention), the sound (messageSound.ts:36), the thread-follow filter (threadAllows) and
suppress_everyone (messageMentionsMe delegates to it). Reasoning against the server:
- The wire list is authoritative on every live path. Every
message_createbroadcast is a message loaded back from the database (GetMessageinpost.gopublishCreated/PublishMessage,webhook/execute.go:227,bot/interactions_respond.go:191,poll/service.go:77, outbox), andloadMentionsfillsMentionsfrommessage_mentions(messages.go:135-148, initialised to[]). The poll's empty-list fallback is only the author's own HTTP response, not broadcast. - Server rule for a mute (
readstate.go:47-49,push/recipients.go:177): addressed = a reply to me, or amessage_mentionsrow of kind name (0) or role (2). The server stores kind 0 only for listed users whose name is in the text (queryMentionedCASE) and kind 2 for listed role members (addRoleMentions). So "listed AND (text names me OR a mentioned role is mine)" is the server rule, with the exception in N1. - Role mention over the recipient cap:
capRecipientsdrops users but keepsRoles, somention_rolesstill lists my role while I am not inmentions. Before the fix the client counted it, the server did not; after the fix both do not. Same for the 100-name cap (maxMentionNames) and forCanView. - Older server without the list:
Array.isArrayis false, the text rule stays (probe P3).
Own probes (not in the commit's tests; scratch test in the worktree, deleted):
| Probe | With fix | Fix reverted |
|---|---|---|
P1 muted, @Dev ping, mention_roles: [myRole], mentions: [] (role dropped by the cap) | badge 0, no notification | fails (badge 1) |
P2 muted, same with mentions: [me] | badge 1 | passes |
P3 muted, hey @max, no mentions field (older server) | badge 1 | passes |
P4 muted, @all @Dev deploy, mention_roles: [myRole], mentions: [me] | badge 1 (see N1) | passes |
P5 suppress_everyone on, level mentions, webhook build failed, @max, mentions: [] | badge 0, no notification | fails |
P6 level mentions, same message (control, unchanged path) | badge 0 | passes |
P7 followed-threads known, reply in an unfollowed thread with unresolved @max | no notification | fails |
So the cause is removed for every variant listed, including role mentions, thread follows and suppress_everyone, which the commit's tests do not cover.
2. Recorded tests
web/src/stores/chatMentionPolicy.test.ts: the two it.fails are plain it now; three cases added (live
badge equals the badge after a reload, @all alone keeps the mute, a reply to me breaks it). With the
production change reverted (notifications.ts from 3be5cb096): 3 failed, 13 passed (the two H-L3-05 cases and
the reload case fail, as the commit message says). With the fix: the five files around it
(chatMentionPolicy, chatThreadFollow, chatComfort, chatLogic, messageSound) 68 passed.
chatThreadFollow.test.ts fixture (lead's question): the reply hello @me came from an ordinary member
(fixtureId(9)) to the signed-in user whose username is me, with mentions: [] taken from
messageFixture's default. The server never sends that: an ordinary member's @me resolves (active user,
can view the channel; the refuter showed the server's parser is not Markdown aware), so the list holds me.
The new fixture (mentions: [me]) is the server's real payload. It is not a weakened test: the case still
checks that a name breaks through the thread filter, and P7 checks that an unresolved name does not.
3. Does it introduce a defect
No behaviour change beyond the finding. No API change (no wire field). messageMentionsMe (unmuted path) is
unchanged. Cost: one Array.includes per message.
B1 (BLOCKER): the new test does not typecheck
web/src/stores/chatMentionPolicy.test.ts:97 builds reply_to: { id, user_id, username, display_name, content } without deleted, which ChatReplyPreview requires (internal/chat/core/message_types.go:166,
json:"deleted"). npx vue-tsc --noEmit -p tsconfig.tests.json fails with TS2322 at 97:50 (1 error; the
app config passes). This config is part of npm run typecheck, so CI is red. Fix: add deleted: false.
4. Gates
ESLint on the three touched web files: clean. vue-tsc app config: clean; tests config: B1. English check: pass.
notifications.ts 124 lines, test 100 lines. Commit type fix. doc/architecture.md gains a paragraph that
matches the new behaviour.
5. The residual the author reports ("@all @Role"), and other mismatches
N1: "@all @Role" is a real server defect, the documents support the client; worth its own finding
addRoleMentions (internal/chat/mentions/mentions_roles.go, the have[id] skip) leaves a role member that
@all/@here already put in the set at kind group (1). The muted badge (readstate.go:49, kind <> group)
and the push (namedMentions, push/recipients.go:177) then do not count them, so @all @Dev keeps the mute of
a muted member of Dev, while @Dev alone breaks it. doc/architecture.md:594-597 ("addressed to the member
(by name, through a role they hold, or as a reply)") and doc/upgrade.md:1041-1045 say the member is
addressed through the role. The client (before and after this fix, probe P4) counts it, so live badge and
reload disagree in the other direction. Code-read, not run. Suggested severity S4 (needs a sender who may
mention everyone and both mentions in one message; effect: a missed notification, plus a live/reload
disagreement). Server fix direction: when a role member is already in the set at kind group, raise the kind to
role instead of skipping. Not this fix's job; the commit message states it honestly.
N2 (candidate finding): webhook/bot with user mentions off still addresses a named member when the message also says @all
mentionSkipped (internal/chat/mentions/mentions.go) does not skip a user that @all reaches, and
queryMentioned gives kind 0 (name) to every listed user whose name is in the text, whatever the policy. So
from a webhook with allow_user_mentions off but allow_everyone on (or a bot with allowed_mentions.parse: ["everyone"] only), @all @max stores max as a name mention and breaks max's mute (badge after reload, push,
and the client agrees). doc/webhooks.md:34: off "turns every member mention into plain text". Code-read, not
run. S4 (unusual configuration). Pre-existing, not caused or changed by this fix.
N3: 100-name cap with @all
A message with @all, 100 other names and then @max: the server reads only 100 names (capNames), so max is
listed through @all (kind group) and not counted; the client sees @max in the text and counts it. Live
badge 1, reload 0. Needs MentionEveryone and 101+ names; noted, not worth a finding.
6. Not checked (H-L3-05)
Server push and read-state queries were read, not run. The desktop client (it loads the web bundle). No end-to-end run with a real webhook.
H-L3-06
1. Does it remove the cause
Cause: raw Markdown cut to 200 characters before stripping. The fix adds mdplain.Snippet (Go,
internal/mdplain/snippet.go) and markdownSnippet (web, web/src/shared/lib/markdown/snippet.ts): a message
of at most 200 code points is returned as it is; a longer one is reduced to plain text (spoilers as a
private-use mark), white space collapsed, cut to 199 plus "…", then escaped back into Markdown with each
spoiler written as ||…||. All sites named by the finding and the refuter use it: reply quotes
(messages_scan.go:35; MessageSelect reads rm.content), the client's edit path (replies.ts
markPreviewEdited), search roots (search_roots.go:53), reminders (later/visibility.go:55, which feeds the
list, the in-app notice and the reminder push), plus the conversation list (dm.go:202 and the client's
lastMessageOf), which the finding missed. Saved items, forwards, message pushes and OS notifications
already stripped the whole message (refuter's table), confirmed.
Against its own parser on each side the fix holds:
- Go deep sweep, run myself:
PLAIN_SWEEP=1 go test -run TestSnippetNeverShows ./internal/mdplain-> "100000 messages, 13519 cut, 0 leaks". Round-trip test (20,000 strings) passes. - Web deep sweep, run myself:
PLAIN_SWEEP=1 PLAIN_SWEEP_N=100000 npx vitest run plainSpoiler.test.ts snippet.test.ts replies.test.ts src/dm: 8 files, 126 tests passed (the sweep asserts zero leaks; the reporter does not print its console line for a passing test). - My own web generator (a richer alphabet:
+,2),-#, entities,<@id>,<:emoji:id>,<t:…>, autolinks, ZWJ families, flags, combining marks, surrogate pairs, fences, escapes; 100,000 messages, 85,450 cut): 0 snippets whosepreviewTextshows a hidden canary, 0 whoserenderMarkdowncontains one, andpreviewText(snippet)equals the expected cut plain text in every case.
But the server's snippet is built by the Go parser, and every client now shows what the Go parser decided. That is B2.
B2 (BLOCKER): the server's parser now decides what web clients show, and it disagrees with the web renderer
Before the fix, the web client stripped the (raw) reply quote with the web parser, the same parser that renders
and hides the spoiler. After the fix, for a message over 200 characters, the server reduces it to plain text with
mdplain.Plain and the client only reads that back. The commit's premise ("any reader that strips it with this
flavour ... gets exactly the cut plain text") holds only if mdplain.Plain hides exactly what the web
renderer hides. It does not. Each side's sweep checks the side against itself, so neither can see this.
Cross-check (my generator, 100,000 messages; oracle: the web previewText of the whole message, which the
commit's own test ties to the renderer):
- the Go snippet shows text the web renderer hides in 940 messages;
- the web reply preview leaked before the fix in 3,267 messages, after the fix in 940: 3,061 fixed, but 734 messages that were hidden before now leak;
mdplain.Plainof the whole message (the message push, pre-existing) disagrees in 978.- Go and web snippets differ for 2,199 of the 100,000 (so a reply quote also changes text when the client rebuilds it after an edit).
One class, minimised: a precedence difference between emphasis and code spans when the emphasis spans a
paragraph break. The web parser lets the code span win, mdplain the emphasis, so the || inside the code
span pair up differently and text between spoiler markers comes out visible. Minimal input and its outputs on
both sides are in a separate, non-public note for the coordinator (this fix is not released).
Not all 734 are classified; other classes may exist.
Concrete failing scenario: a member writes a long message (over 200 characters) of that shape with a spoiler inside the first 200 characters. Before the fix every reader's reply quote hides the spoiler; after it, every web and desktop client shows its text in every reply quote, the conversation list, the search root and the reminder list. Same S3 effect as the finding, new trigger.
Fix direction: make mdplain.Plain agree with the web parser (here: code spans before emphasis), and add a
cross-implementation check, for example a generated corpus checked into testdata with the web renderer's hidden
set (or the web previewText) recorded, checked by a Go test against Plain and Snippet. The shared
snippet.json has 14 hand-written vectors and does not catch it. This also fixes the pre-existing push leak,
which deserves its own finding (see N6).
2. Recorded tests
Go internal/later/snippet_spoiler_test.go (inverse assertion) became a plain check on mdplain.Snippet; web
plainSpoiler.test.ts it.fails became plain it; new snippet_test.go, snippet.test.ts,
testdata/snippet.json, and an integration test TestAReplyQuoteHidesASpoilerThatClosesAfterTheCut.
Revert check, done: Snippet and markdownSnippet replaced by the raw cut (textcut.Runes(content, limit-1)+"…" / clampChars). Go: TestSnippetAgreesWithTheSharedFixtures, TestSnippetNeverShows... and
internal/later TestASnippetHidesASpoilerThatClosesAfterTheCut fail. Web: 16 tests fail (plainSpoiler's two
cases and the fixture vectors). Restored afterwards.
Run with the fix: go test -p 2 ./internal/mdplain ./internal/later ./internal/chat/messages ./internal/chat/search ./internal/chat/dm ./internal/push ok; with -tags=integration (OrbStack):
./internal/later ./internal/chat/dm ./internal/chat/search ./internal/chat/messages ok, and ./internal/server -run 'Reply|Search|DM|Dm|Later|Remind|Snippet|Quote' ok, including the new integration test.
3. Does it introduce a defect (other than B2)
API (lead's question)
doc/api-stability.md: field types are unchanged, so the change is not breaking by its rules; it is a change of
what a documented string contains, which the commit states in doc/upgrade.md and the OpenAPI descriptions
(make openapi-check passes; npm run contracts:check passes; event golden tests pass).
- Old web clients and the desktop client (which runs a web bundle): I copied the strip code of v1.0.0 and
v3.0.0 (
web/src/lib/markdown*,replies.ts) and ran theirpreviewTexton 85,450 new snippets: identical to the new client's output in every case. No stray backslashes: every character the escaper prefixes is a symbol both old parsers already treat as escapable (any non-alphanumeric, non-space character), and||…||reads as a spoiler there. - Bots, outgoing webhooks, third-party clients that render these fields as Markdown: see S3.
S2 (SHOULD-FIX): worst-case cost on read paths is about 80 times the measured figure
Benchmark confirmed: BenchmarkSnippetOfALongMessage 27.5-28.5 µs/op, 36.7 KB, 261 allocs (M-series Mac,
3 runs). But mdplain.Plain is quadratic on some inputs, and the four queries now run it on up to 4,000
characters per row, per reader, per load (before: 200). Measured Snippet on 4,000-character inputs:
[a]( repeated 1.6 ms, [ repeated 0.86 ms, mixed openers 0.1 ms. Plain on [a]( repeated:
500 chars 44 µs, 1,000 176 µs, 2,000 707 µs, 4,000 2,260 µs (quadratic). Anyone who can post can make
a message page that quotes 50 such messages cost about 100 ms of server CPU for every reader who opens it, and
the same on the DM list and search roots. Before the fix this cost was paid once per message by the push only.
Not a blocker (bounded by MaxMessageLen and page size), but the commit's "1.3 ms per page" is the typical case,
not the bound. Options: make the link-opener scan in mdplain linear (it already memoises misses for other
openers), or build the snippet once at write time, or at least memoise per quoted message within a query.
S3 (SHOULD-FIX): document that a snippet can exceed 200 characters and is no longer Markdown from the message
- The escaped snippet is longer than its plain text: up to about 5 characters per plain character (each
spoiler mark becomes
||…||, each symbol gets a backslash). Measured: 996 characters for a long run of|, 399 for runs of ``,*,<. The old OpenAPI text said "at most 200 characters" forlast_message.text, the search rootsnippetand the remindersnippet. api-stability.md counts a larger maximum as not breaking, butdoc/upgrade.mdshould say it, and the new descriptions ("the first 200 characters of its plain text as Markdown") can be read as a 200 bound. - A reader that renders the snippet as Markdown (a bot, a third-party client; not ours: every web site strips
it) now sees text the original showed as code: a URL that was inside a code span becomes a live autolink
(
`https://x.example`gives a bare link; the link text is the URL itself, so no disguise), and an@everyoneinside code becomes a mention pill (display only: the server's notification is decided on the message, and its parser already treats code-span mentions as real). No hidden text, no new notification, and custom emoji,<@id>and<t:…>tokens are neutralised by the escaped<.doc/upgrade.mdnames bots; it should also name outgoing webhooks (message_createdeliveries carryreply_to.content).
Escaping, edges (lead's questions)
- Can a crafted message make the snippet contain hidden text, a link or a notifying pill: against the web parser, no (B2 aside, which comes from the Go parser, not the escaper); links and pills only as in S3. A spoiler in the snippet holds only "…": clicking it reveals "…".
- 200 and 201 characters: 200 code points is returned raw (spoiler intact, hidden by the reader); 201 goes
through plain, ends
… ||…||and reads "Spoiler". Checked in Go. - Surrogate pairs, ZWJ sequences, flags, combining marks at the cut: both
textcut.RunesandclampCharsback up to a character boundary; checked with emoji, a ZWJ family, a flag pair ande+ U+0301 at the cut: no broken character. (an entity for the private-use mark) stays literal in both parsers; a U+E000 in the message is dropped before marking (test exists).- A long message whose plain text is empty (for example a long
[) gives an empty snippet; withomitemptythe reply quote then has nocontent. Harmless.
4. Gates
gofmt clean; go vet with and without -tags=integration clean on touched packages; make complexity-check
exit 0; internal/repo tests pass; ESLint clean on the seven touched web files; vue-tsc app and tests configs
clean; English check pass. New files: snippet.go 69 lines, snippet.ts 49. Commit type fix. No allowlist entry,
no raised limit, no skipped test.
5. The two never-stripped sites (lead's question)
N4: event comment delete dialog (candidate finding, S4)
web/src/events/components/EventComments.vue:62 puts c.content.slice(0, 160) raw into the confirm dialog,
while the comment itself renders through MarkdownContent (spoilers hidden). Whoever deletes a comment (its
author or a moderator) sees the spoiler text and the Markdown source. The forum's equivalent
(useForumPostActions.ts:67) uses previewText(...). Also slice cuts UTF-16 units and can split a surrogate
pair. Small, real, outside H-L3-06's mechanism; worth an S4 finding with a one-line fix
(previewText, then a character-aware cut).
N5: wiki search snippets (harmless or S4)
internal/wiki/store_search.go:48-57 returns a raw 160-rune window around the first match and
WikiSearch.vue:52 shows it as text, so a ||spoiler|| near the match shows its text and the Markdown markers.
Everyone who gets the hit can read the page. Arguably a raw search excerpt, not a plain form. I would file it
together with N4 as one S4 "raw excerpts show spoilers" finding, or leave it.
N6: pre-existing push leak (candidate finding, S3)
The B2 disagreement already reaches the message push today: mdplain.Plain of the whole message shows text the
web renderer hides in 978 of my 100,000 generated messages (push/payload.go:222). Not caused by this fix,
but the same fix closes it. Suggest a separate finding so the cross-implementation property gets its own test.
6. Not checked (H-L3-06)
End-to-end push delivery; the native parts of the desktop client (only its web bundle's strip code, by copying
v1.0.0 and v3.0.0); whether every insert path enforces MaxMessageLen (the cost bound in S2 assumes it); a
classification of all 734 regressions beyond the one minimised class; the web markdownSnippet cost on
pathological input (it runs only on the client's edit and conversation-list paths).
Summary
- H-L3-05: MERGE AFTER FIXES. B1 test typecheck red (
deletedmissing at chatMentionPolicy.test.ts:97). N1 "@all @Role" real server under-count, documents back the client, own finding (S4). N2 user mentions off plus @all still addresses by name (candidate S4). N3 100-name cap mismatch (note). - H-L3-06: MERGE AFTER FIXES. B2 Go parser now decides what web clients show and disagrees with the web renderer: 734 of 100,000 generated messages newly leak (3,061 fixed). S2 quadratic worst case on read paths (about 2 ms per snippet). S3 document longer snippets and Markdown rendering by bots and webhooks. N4 event comment delete dialog shows spoilers (candidate S4). N5 wiki search raw window. N6 pre-existing push leak from the same disagreement (candidate S3).
Second round
New commits reviewed: 423118a9a (fix-h-l3-05), 5a763ac81 and 70b7dc072 (fix-h-l3-06). Fetched first; fresh
detached worktrees with npm ci, removed afterwards; probe files and generated corpora deleted. Nothing
committed or pushed.
| Fix | Verdict |
|---|---|
| H-L3-05 | MERGE (B1 closed; the coordinator folds the test-only fix: commit) |
| H-L3-06 | MERGE (B2, S2, S3 closed; NOTEs N7 to N9 only) |
H-L3-05: B1 closed
423118a9a adds deleted: false to the reply preview at chatMentionPolicy.test.ts:97; nothing else changes.
vue-tsc --noEmit with tsconfig.app.json: exit 0; with tsconfig.tests.json: exit 0 (it failed before with
TS2322). chatMentionPolicy and chatThreadFollow tests: 16 passed. The commit type is fix: for a change that
only touches a test; it should be test:. As agreed, the coordinator folds it into 59aeb545b.
H-L3-06: B2 closed
5a763ac81 replaces the server's mdplain with a port of the client's parser (blocks, inline, link, plain,
working on UTF-16 units). Where only a browser can judge a link host and that host could move a spoiler, or
where the work bound runs out, it fails closed: everything between the first and the last || counts as
hidden. Both are explained in plain.go's package comment.
My own cross check, on seeds neither of us used
This is the same generator and oracle that found the 734 regressions in the first round: the web
previewText of the whole message, plus the renderer-HTML verdict from the new fixture, whichever hides more.
Added this time: visible canaries in plain text, so that dropped visible text can be measured. Seeds 301, 302
and 303 (100,000 messages each, about 85 % longer than 200 characters), plus 100,000 messages without any |
(seed 304). Main's server side is a copy of the old mdplain (7a91dd9fa). Main's reply preview is the web
previewText of the first 200 code points.
| Corpus | New push leaks | New snippet leaks | Main push leaks | Main reply-preview leaks | 7a91dd9fa snippet leaks | Regressions (push / snippet) |
|---|---|---|---|---|---|---|
| seed 301 | 0 | 0 | 1,331 | 12,579 | 1,287 | 0 / 0 |
| seed 302 | 0 | 0 | 1,310 | 12,660 | 1,266 | 0 / 0 |
| seed 303 | 0 | 0 | 1,309 | 12,642 | 1,263 | 0 / 0 |
| seed 304, no ` | ` | 0 | 0 | 0 | 0 | 0 |
The two oracles never disagreed in either direction.
The oracle cannot pass by hiding everything
In my corpora, the new mdplain.Plain equals the web previewText character for character (after collapsing
white space) in all 400,000 messages. The Go snippet's plain form equals the web snippet's in all 400,000.
Visible canaries the web shows: 1,519,026. Visible canaries the new form drops: 0. Characters lost: 0 of 86.8
million.
Main's old mdplain, by contrast:
- dropped visible text in about 1.9 % of the messages with spoilers (1,831 to 1,898 per 100,000);
- differed from the web in 5,744 of the 100,000 messages without
|.
So the claim that "messages without || are unaffected" holds in the sense the lead asked about: nothing in
them is blanked or hidden. Their push text does change for about 6 % of my generated messages without |.
It changes toward the web renderer (N7).
My generator has no hosts that only a browser can judge, so nothing failed closed there. To look at the
fail-closed share, I ran the author's generator (plainCross.fixture.ts) on new seeds 7101 to 7103 (100,000
messages each):
- the author's
TestPlainHidesWhatTheRendererHidesgives leaks 0, failClosed 908 and 870, plainDiffers 0, snippetDiffers 0; - my analysis of the same records gives leaks 0 and regressions 0; main's push leaked in 9,794 to 9,960 messages.
- About 0.9 % of messages differ from the web (898 to 921), and nearly all of them contain
||. That is the fail-closed share.
What a reader sees in a fail-closed message (N8):
- the text before the first
||, one "Spoiler", then the text after the last||; - in those messages, the median loss is 52 to 54 % of the visible text (p90 86 to 87 %, max 98 %); the median is about 140 characters, the max about 520;
- about 0.75 % of all messages lose at least one visible canary.
The triggers are a masked link or an autolink whose host the browser must judge: non-ASCII, punycode, a percent sign in the host, or IPv6. A bare URL fails closed only when the uncertain reading could move markup. For real messages that means a link like that plus two or more spoilers with text between them. A message with a single spoiler loses nothing.
mdplain's callers
grep mdplain. outside the package finds exactly six callers:
push/payload.go:222(plainLine);onboarding/sharepage.go:62(the public share page's description of the welcome text);messages_scan.go:35,search_roots.go:55,dm.go:204,later/visibility.go:57(the snippets).
There is no search indexing, mail text or other notification text that uses it. All of them now get the web's reading: the same visible text except in the fail-closed case. The share page also loses nothing visible, because its spoiler word is "…" and fail-closed only affects text between spoilers.
Tests I ran:
- unit tests:
./internal/mdplain ./internal/push ./internal/onboarding ./internal/later ./internal/chat/... ./internal/repoall ok; - integration tests (
-tags=integration):./internal/later ./internal/chat/dm ./internal/chat/search ./internal/chat/messages ./internal/push ./internal/onboardingok, and./internal/serverwith-run 'Reply|Search|DM|Dm|Later|Remind|Snippet|Quote|Push|Share|Onboard'ok.
Golden data and fixtures:
plain.jsongains 12 vectors. No existing vector changed.snippet.jsonchanges one vector, by the escaped:of S3, and adds one.plain_test.goonly renames a helper test (hasPrefixFold to hasAtFold).- No push or onboarding expectation was edited.
So no golden data was adjusted to hide a change.
Work bound (S2 closed)
Here is what I measured for Plain and Snippet at 4,000 and 6,000 characters. 6,000 is the parser's own
limit; above 6,000 UTF-16 units both sides leave the text unparsed, as before.
- My first-round worst cases at 4,000:
[a](116 µs (was 2,260 µs forPlain, 1,635 µs forSnippet),[73 µs (was 834), mixed openers 130 µs. - Seven new hostile shapes of my own, at most 126 µs at 4,000 and 187 µs at 6,000:
- backtick runs of growing length;
<https://openers;- failing bare URLs;
- deeply nested masked links;
- escaped pipes;
- quote and list markers;
- IDN masked links between spoilers.
- Steps never exceeded 30,000 against a budget of 260,096 (4,000) and 388,096 (6,000). Time grows linearly from 4,000 to 6,000.
- A dense ordinary message: 62 µs (66 µs as a snippet), against 27 µs for the old mdplain. That matches the author's 0.026 to 0.064 ms.
The new TestPlainWorkIsLinearInTheMessage pins twelve more hostile shapes.
S3 closed
The escaper now also escapes the : of http(s)://. In the web renderer, the snippet of a long message
contains no <a for each of these: a URL in a code span (lower and upper case), an autolink in a code span,
a masked link in a code span, http://, and a message with a real link and a bare URL. Old clients (v1.0.0,
v2.9.12, v3.0.0) treat : as escapable (/[^0-9A-Za-z\s]/), so https\:// reads as https:// there.
MaxSnippetUnits = 996 is pinned by a test that also requires the all-spoiler case to reach exactly 996.
doc/upgrade.md now names outgoing webhooks, states the 996-character bound and explains the fail-closed
rule.
Push in both languages
internal/push/payload_hidden_test.go runs the 1,000-record corpus through clean(plainLine(..)) for en
and de and fails on any hidden canary. It passes.
Nightly workflow (70b7dc072)
- Triggers:
scheduleandworkflow_dispatchonly. - Permissions:
contents: read. - Checkout uses
persist-credentials: false. - Actions are pinned to full SHAs, the same SHAs the other 26/8/12 uses in the repository's workflows (checkout v7.0.1, setup-node v7.1.0, setup-go v7.0.0).
- No secrets and no untrusted input. The seed is
1000 + GITHUB_RUN_NUMBER, expanded by the shell from the environment. timeout-minutes: 60.- The npm cache is used as in
sim-sweep.yml. - Expected run time: locally, writing 100,000 records took 8 s and the Go check 3.4 s, so 200,000 is a few
minutes plus
npm ci. - AGENTS.md lists the workflow.
Gates
- gofmt clean.
go vetwith and without-tags=integrationclean.make complexity-checkexit 0.internal/repook.- English check passes.
make openapi-checknot stale.- ESLint on the three web files clean.
- vue-tsc app and tests configs exit 0.
- Web markdown, replies and dm tests: 452 passed, 4 skipped (env-gated).
Production files are at most 339 lines. The only lines over 150 characters are two first-round lines in
snippet_test.go. The split functions read normally: no packed statements and no long one-liners.
NOTEs
- N7: push texts change for ordinary messages. The parser swap changes the push text of messages without
spoilers too. In my generated corpus without
|, 5.7 % of messages read differently from main. Every such change brings the text to exactly what the web shows, and nothing visible is lost. Release notes could say "push texts now read like the message view" rather than "unaffected". - N8: what fail-closed costs. About 0.9 % of the author's generated messages fail closed, and they lose about half their visible text to one "Spoiler". In real messages this needs a link with a host only a browser can judge, plus two or more spoilers. The cost is acceptable for a rule that cannot leak. It could be narrowed later by failing closed only around the spoiler pairs the uncertain link can reach.
- N9:
423118a9acommit type. It isfix:for a test-only change. The coordinator folds it.
The first round's candidate findings stand unchanged: N1, N2 and N4 to N6. N6, the pre-existing push leak, is fixed by 5a763ac81 together with B2, so it now needs only a record, not a fix.
Not checked (second round)
- The 0.9 % fail-closed set is not classified beyond the triggers in the code.
- No end-to-end push delivery.
- The nightly workflow was not run on GitHub (only its steps, locally).
Third round
One commit: b01ffa11a on msch128/hunt-harness-1 (parent f0714dd85), fetched first. It ports the client's new
authority rule (linkAuthority.ts plainAuthority: no user info, percent escapes that decode as UTF-8, no
\p{Cc}/\p{Cf} in the authority as written or decoded) into internal/mdplain/host.go, and makes Go
decide hosts that are ASCII once percent-decoded. It also adds the drift guard (hrefParity.test.ts, which
writes testdata/hrefs.json, checked by TestSafeHrefAgreesWithTheClient) and extends the cross generator.
Fresh detached worktree with npm ci, removed afterwards together with all probes and corpora. Nothing
committed or pushed.
Verdict: MERGE (H-L3-06 with this commit; no BLOCKER, no SHOULD-FIX; NOTEs N10 and N11).
My cross check on this state
Same generator, oracle and comparison as the second round. Seeds 401 to 403 (100,000 messages each). Seeds 404 and 405 (100,000 each) are link-rich: 40 % of the pieces are taken from 32 link shapes:
- user info (
u:p@,a.example@b.example,@h, user info in masked links and autolinks); - percent-encoded authorities (
%2E,%41,%40,%2540,%00,%2F,%3A,%E2%80%8B,%zz,%C3%A9, the overlong%C0%80, encoded IPv4 shorthand, upper-case hex); - control and format characters raw in hosts (U+200B, U+00AD, U+2060, U+200E, U+0085, U+0007);
- IDN hosts.
"Main" means origin/main's mdplain for the push. For the reply preview it means LEFT 200 followed by this
branch's web strip.
| Corpus | New push leaks | New snippet leaks | Main push leaks | Main reply leaks | Regressions (push / snippet) | Oracle disagreements |
|---|---|---|---|---|---|---|
| seed 401 | 0 | 0 | 1,626 | 12,270 | 0 / 0 | 0 |
| seed 402 | 0 | 0 | 1,654 | 12,320 | 0 / 0 | 0 |
| seed 403 | 0 | 0 | 1,704 | 12,151 | 0 / 0 | 0 |
| seed 404, link-rich | 0 | 0 | 62 | 6,561 | 0 / 0 | 0 |
| seed 405, link-rich | 0 | 0 | 82 | 6,470 | 0 / 0 | 0 |
Ordinary corpora (401 to 403):
- the new plain form and snippet equal the web's character for character in all 300,000 messages;
- the new form drops 0 of 1,230,045 visible canaries;
- main dropped visible text in about 2.3 % of these messages.
Link-rich corpora (404, 405):
- about 16 % of messages differ from the web, nearly all of them by failing closed;
- 12.4 % lose at least one visible canary, about 7 % of the visible characters in all.
I traced that back. Of the 32 link shapes, exactly one makes Go unsure enough to fail closed: a masked link
to a non-ASCII host ([x](https://café.example)). My generator draws it often, and most messages carry
several spoilers. None of the percent-encoded, user-info or control/format shapes makes Go unsure any more.
Before this commit, any % in a host did. So the fail-closed share depends on how common masked IDN links
are, as in N8. The 0.4 % the author measured is plausible for their generator.
In 10 link-rich messages without ||, the Go text differs from the web's because a masked IDN link is shown
as source instead of its label. That is the documented "unsure: may show text differently, never what is
hidden" case (N10).
Can Go now be sure and wrong?
I recorded the client's safeHref answer for 60,216 targets and compared them with the Go port wherever it
says it is sure. The targets: 71 hand-written authorities, each under http://, https:// and HTTPS://,
plus 20,000 random authorities built from % %2E %2F %40 %3A %25 %00 %41 %C3%A9 %E2%80%8B %7F %5C %5B %5D @ : [ ] \ 0x, U+200B, U+00AD, U+0085 and letters, digits and dots.
The hand-written cases:
%2540and%252E(double encoding);%00;- encoded dots, including
%2E%2E, a trailing%2Eand lower-case%2e; - encoded
/,\,@,:,#,?,%,[,]and|; - encoded port separators, and ports with
%,+,-, a space and leading zeros; - encoded IPv4 shorthand:
%31%32%37.1,0x%37f.1,%30X7F.1,127.%31, 4294967295 and 4294967296 encoded, anda.%31; - encoded DEL, TAB and LF;
- NBSP, soft hyphen, BOM, ZWSP and emoji percent-encoded;
xn%2D%2D;- backslash before
@; - user info, written and encoded.
Result: 0 answers where Go is sure and wrong, and 3,672 left to the browser. The decoding mirrors the URL parser:
- an encoded
@,/,:,%,\or[decodes to a forbidden domain code point, so the host is refused, as the browser refuses it; %2540decodes once to%40and is refused for its%;%00is refused by the Cc rule;- encoded dots and digits feed the same label and IPv4 logic as written ones;
- the port is split on the raw
:before decoding, as in the URL parser.
Cc and Cf, Go against JavaScript
linkAuthority.ts tests /[\p{Cc}\p{Cf}]/u. Go's hidden tests unicode.Cc and unicode.Cf. I compared
the two over every code point:
- Go 1.27.4 has Unicode 17.0.0, Node 24.12 has Unicode 16.0 (ICU 77.1).
- Both sets have 235 code points and are identical, with no code point in only one of them.
A user's browser may run an older Unicode version, where a few code points that are Cf now (for example the Egyptian format controls U+13439 to U+1343F, added in 15.0) are unassigned. There Go refuses the link as "hidden" while the old browser's regex does not match. But an unassigned code point is disallowed by that browser's IDNA, so it refuses the host too, and both readings agree that it is no link. In a path or query the rule does not apply on either side. No disagreement follows (N11).
The drift guard
I mutated linkAuthority.ts two ways:
- dropped the
authority.includes('@')test; - narrowed
HIDDENto\p{Cc}.
Each time hrefParity.test.ts failed ("the recorded answers are the client's"). After HREF_UPDATE=1,
Go's TestSafeHrefAgreesWithTheClient failed: 3 or more mismatches after mutation 1, such as
safeHref("http://@h.x/") = false, the client says true, and 8 after mutation 2. Restored afterwards. So a
change to the client rule cannot pass both sides unnoticed.
Tests and gates
- Full vitest, 3 workers, once: 662 files passed, 2 skipped; 12,025 tests passed, 8 expected fail, 98 skipped; 1 min 54 s.
- Go:
./internal/mdplain ./internal/push ./internal/onboarding ./internal/later ./internal/chat/... ./internal/repo ./internal/automodok. - gofmt clean,
go vetclean,make complexity-checkexit 0, English check passes. - ESLint clean on the two web files; vue-tsc tests config exit 0.
host.gois 197 lines.cross.jsonlwas regenerated with the extended generator. That is not a hidden golden change: the web test requires it to equal what the renderer says, record for record.doc/upgrade.mdnow names percent-encoded non-ASCII hosts and punycode for the fail-closed rule.
NOTEs
- N10: masked IDN links read as source. In a message without spoilers, Go may show a masked link to a non-ASCII host as its Markdown source where the web shows the label. This is by design (text may differ, nothing hidden is shown). It affects only push text and snippets.
- N11: older browsers' Unicode tables. The Cf set of the user's browser may be older than Go's. Code points that are Cf in new versions are unassigned, and so refused by IDNA, in old ones. Both sides then refuse the link.
Not checked (third round)
- Real browsers other than Node's URL implementation; the guard and my comparison both use Node.
- IDNA mapping of non-ASCII hosts: Go leaves those to the browser by design.