Fix review: H-L3-09 (wiki titles: one Unicode-aware key on the server and in the app)
From mnema-talk v3.3.13, rendered on this site.
Reviewer: fix-l3-09 review (phase 3), read-only. Fix: branch msch128/fix-h-l3-09, commit 200b1284c ("fix: wiki titles compare by one Unicode-aware key on the server and in the app"), reviewed on its own against its parent. One temporary worktree, removed when done; nothing pushed, nothing changed in the repository.
Verdict: MERGE AFTER FIXES
One SHOULD-FIX (a performance regression on the page view and the editor preview), no blocker. The fix removes the cause the finding and the refuter describe (two rules, no normalisation), for both variants (NFC/NFD look-alikes, the U+0130 disagreement), and the two key functions agree on every code point and on 300,000 random multi-character strings under the Node version CI uses.
Cause removed?
Yes. The server's titleTaken (internal/wiki/title_key.go:50) and the client's resolveWikiLink
(web/src/wiki/lib/wikiTree.ts:62) both compare NFC(simpleLower each code point(NFD(title)));
titles are stored NFC (cleanTitle, title_key.go:38, used by cleanText and by RestoreRevision).
Every writer of wiki_pages.title goes through checkTitle (Create, Change via save, RestoreRevision via
save, Undelete); there is no other writer in internal/ (grep for INSERT INTO wiki_pages and
UPDATE wiki_pages SET ... title).
Do the two key functions agree? (own exhaustive comparison)
Own harness (outside the repository): the Go function copied byte for byte (same golang.org/x/text
v0.43.0, Go 1.27.4: unicode.Version 17.0.0, norm.Version 17.0.0) and the TypeScript function copied
into a plain Node script. Each prints the key of every code point U+0000..U+10FFFF except surrogates
(1,112,064 inputs), and of 300,000 random strings of 1 to 8 code points (3/4 drawn from a pool of
special-casing letters, final and medial sigma, U+0130/U+0131, titlecase digraphs, Kelvin/Angstrom/Ohm
signs and other canonical singletons, composition exclusions such as U+0958, U+FB2A, U+2ADC, U+1D15E,
combining marks of many combining classes in random order, Hangul jamo L/V/T and syllables, astral cased
scripts, the Unicode 16 and 17 additions; 1/4 uniform over all planes).
| Comparison | Single code points that differ | Multi-character strings that differ |
|---|---|---|
| Go 1.27.4 vs Node 26.13.1 (ICU 78.3, Unicode 17; the version every CI job uses) | 0 | 0 of 300,000 |
| Go 1.27.4 vs Node 24.12.0 (ICU 77.1, Unicode 16) | 28 | 15,742 of 300,000, every one containing one of the 28 |
The 28 are exactly the case pairs new in Unicode 17: U+A7CE, U+A7D2, U+A7D4 (Latin Extended-D) and U+16EA0..U+16EB8 (Beria Erfe capitals). Go maps them to their lower case, Node 24 leaves them. No normalisation difference at all (every difference is a case mapping). This matches the author's claim. All 28 lie above U+A000, so the vector sweep (U+0020..U+9FFF) does not contain them.
What happens in production when server and browser disagree
The server (Go, Unicode 17) is the only place that enforces uniqueness; the browser only resolves links, with whatever ICU the reader's browser ships.
- Browser older than the server (the 28 characters today, in any browser below Unicode 17): the server
folds more than the browser. A page "X" whose title holds a new capital, linked as
[[x]]with the small letter, shows as a missing link (dotted underline, web/src/wiki/lib/wikiLinks.ts:18) for that reader; creating it is refused by the server as a taken title. No duplicate can arise: two titles the browser keeps apart but the server joins cannot both be created. - Browser newer than the server (a future Unicode in the browser before Go updates): the browser joins two titles the server keeps apart. Both pages exist (they differ only in the case of a new letter, so they look different); a link resolves to the first in tree order, as for an old shared pair. No duplicate in the server's sense.
- A Go upgrade that adds case pairs turns two existing distinct pages into a shared-key pair: handled
exactly like the pre-fix pairs (both stay, both editable,
savecompares keys). - Nothing worse: no error path, no crash, no permission effect (a link only navigates among pages the reader already sees). Worst case is a link shown as missing for 28 rare characters in older browsers.
Recorded tests: normal tests, fail without the fix
- wikiTitleFold.test.ts:
it.failsis now a normalit; the known-disagreement list is now[]. - Revert run (store_write.go, store_edit.go, wikiTree.ts back to the parent; title_key.go reduced to
titleKeyplus the oldcleanTitleso the tests compile):go test ./internal/wiki/: TestATitleIsStoredComposed fails (3 cases).go test -tags=integration -run TestPagesThatAlreadyShareATitleKeyStayEditable ./internal/wiki/: fails.go test -tags=integration -run TestWikiTitlePairsMatchTheServer ./internal/server: 9 mismatches (the 8 NFC/NFD pairs and the U+0130 pair), as the commit message says.- vitest src/wiki/lib/: 2 fail (the disagreement test and the new resolveWikiLink test).
- TestTitleKeysMatchTheVectors cannot fail on a revert (it tests
titleKeyitself, which the revert keeps); it guards drift between the two implementations, not the store.
- With the fix:
go test ./internal/wiki/ok;go test -tags=integration ./internal/wiki/ok (whole package);go test -tags=integration -run 'TestWiki' ./internal/serverok; vitest src/wiki/lib/ 12 files, 116 passed, 2 skipped, on Node 24.12.0 and on Node 26.13.1.
The specific questions
İstanbul next to Istanbul. TestTitlesAreOneLineAndUniqueIgnoringCase
(internal/wiki/store_integration_test.go:517) is unchanged by the commit and passes, with and without the
fix. It does not contain U+0130. Its "a lookalike is the same title" (line 524) is about a zero-width
space, which renders as nothing; U+0130 renders with a visible dot, so allowing "İstanbul" next to
"Istanbul" does not contradict its intent. The change is documented (doc/wiki.md) and makes the server
agree with what the client resolves, which is the property. Accepted as a deliberate behaviour change.
(Side effect worth knowing, not a defect under the documented rule: [[istanbul]] does not resolve to
"İstanbul"; the rule is locale-independent.)
Race. Not widened, and there is none. Every write path (Create store_write.go:150, inTx used by
Undelete store_write.go:210, Change store_edit.go:96, RestoreRevision store_edit.go:260) takes
pg_advisory_xact_lock(hashtext('mnema.wiki')) (store.go:150) before the check and holds it until
commit, so check and insert/update are serialised. That was true before (the old SQL EXISTS ran under
the same lock) and is true now. Own run: 20 rounds of 4 concurrent Create calls whose titles share one
key (NFC, NFD, upper and lower case): exactly one succeeded per round, the others got errTitleTaken.
The 500-title bound. MaxPages = 500 (internal/wiki/model.go:23) is enforced by checkCount
(store_write.go:126) in Create and Undelete, under the tree lock; deleted pages are excluded from both the
count and titleTaken's query. Rows written before the limit could exceed it, but the cost is linear in
any case. Cost measured (own Go benchmark, 500 titles of ~35 characters plus a 100-character title):
0.70 ms and ~740 KB allocated per check, plus reading up to 500 short rows. It runs only on create,
title change, restore and undelete (rate-limited routes), inside the lock. Acceptable.
Stored NFC on an unrelated edit. The web editor always sends the title
(web/src/wiki/composables/useWikiEditor.ts:29), so the first edit of a page stored in NFD rewrites its
title bytes to NFC. Checked with an own integration test: a text-only edit of an NFD-titled page writes a
revision with the NFC title. Effects: the slug does not change (slugs come from the title only on
create; rename takes an explicit slug); links resolve by key, so no link breaks; history gets a
revision whose title differs only in bytes (see NOTE 3); search is ILIKE on the stored bytes (see NOTE 4).
Restore of an old revision. Goes through cleanTitle (store_edit.go:272) and save, so the title is
stored NFC and checked by key. Own test: restore onto a key held by another page is refused (also in the
author's test). See NOTE 3 for the no-op restore and NOTE 2 for titles that grow under NFC.
Rename onto a key an existing pair holds. Own test: two legacy pages "Xé" (NFC) and "Xé" (NFD), a third page renamed to "XÉ" is refused (errTitleTaken). One member of the pair renamed to another case of its own key ("XÉ") is allowed, which creates no new duplicate. Undelete of a legacy page onto a key now held by another page is refused.
The vector file. internal/wiki/testdata/title_keys.json is pure ASCII (no byte outside 0x20..0x7E
and newline): the writer escapes every UTF-16 unit, so astral characters are valid JSON surrogate
pairs, which Go's encoding/json decodes to the right code point (the Go test passes on them).
Regenerated with WIKI_KEYS_WRITE=1 under Node 24.12.0 and under Node 26.13.1: both outputs are byte for
byte identical to the committed file (46 vectors). Stable across these versions because the sweep stops
below U+A000, where Unicode 16 and 17 have the same case data. A future Node whose ICU adds a case pair
below U+A000 would make expect(v.map(x => x.title)).toEqual(titles()) fail until the file is
regenerated; that is the intended alarm, not a flake.
Findings
SHOULD-FIX 1: link resolution is now ~60 times slower on the page view and on every keystroke in the editor preview
web/src/wiki/lib/wikiTree.ts:62-69. resolveWikiLink computes wikiTitleKey(p.title) (two
normalisations and a per-code-point loop) for every live page, for every [[...]] in the text, on every
render. wikiLinkRule (wikiLinks.ts:11-23) calls it per match, and MarkdownContent recomputes the whole
render whenever content or wiki.live changes (shared/components/MarkdownContent.vue:51-56); the
editor's side-by-side preview (WikiEditor.vue:119) re-renders on each keystroke.
Measured in Node 24 on an Apple-silicon Mac (own benchmark, 500 pages of ~35 characters):
| Text | Before | After |
|---|---|---|
| An index page linking all 500 pages | 2.5 ms | 146 ms |
| 1,000 links to missing pages | 15 ms | 520 ms |
| 10,000 links to missing pages (fits in the 50 KB content limit) | 139 ms | 5,100 ms |
So on a full wiki, typing in the editor of an index page costs ~150 ms per keystroke on a fast laptop
(several times that on a phone), and a page full of [[x]] freezes every reader's tab for seconds.
Fix: compute each page's key once per page list, for example a Map from key to the first slug in list
order built once in wikiLinkRule(pages) (or memoised in wikiViews), and look links up in it. Measured:
1.4 ms per render for the 500-link index page, including building the map. The "first in tree order"
rule is kept by inserting only when the key is not yet in the map.
NOTE 2: titles that grow under NFC are refused at a length the writer did not type
title_key.go:43 checks the length again after NFC (needed: the column is VARCHAR(100), Postgres counts
code points). For composition-excluded characters NFC gives two code points for one (U+0958..U+095F,
U+09DC.., U+FB1D, U+FB2A..U+FB4E, U+2ADC, U+1D15E..). Own test: creating a title of 60 × U+0958 is
refused "title must be at most 100 characters" (before: accepted). A legacy page whose stored title is 60
× U+0958 can no longer be edited from the web app (which always sends the title) until the title is
retyped shorter or in decomposed form; a text-only PATCH still works. Restoring an old revision with
such a title is refused the same way. Keyboards produce these letters decomposed, so this needs pasted
precomposed forms, many of them, near the limit; very unlikely. Possible remedy: in save, keep the
current bytes when the key is unchanged and the NFC form would not fit. Not required for merge.
NOTE 3: a title-only normalisation writes a revision, and a no-op restore is no longer refused
reviseText (store_edit.go:140) and restorable (store_edit.go:297) compare titles by bytes. Own test,
legacy NFD page: an edit with unchanged visible title and text writes revision 2 (title bytes NFC); a
restore of revision 1 (NFD, same text) is then accepted and writes revision 3 that changes nothing
visible, where errNothingNew would be the expected answer. Harmless (history noise, once per legacy
page); comparing cleanTitle(target.Title) in restorable would restore the refusal.
NOTE 4: wiki search does not normalise the query
internal/wiki/store_search.go:29 matches title ILIKE on stored bytes. Titles are now NFC, so a query
typed in NFD finds no title (own test: NFD query 0 hits, NFC query 1 hit). Before the fix the same gap
existed in the other direction for NFD titles, and content is not normalised either, so this is not a
regression of the fix, but normalising the query to NFC would complete it.
NOTE 5: "Titles are stored in Unicode NFC" holds for pages written after the fix
doc/wiki.md: a legacy NFD page keeps its bytes until it is saved (Undelete does not normalise, store_write.go:187-194, own test). The following paragraph on old pages covers the consequence (pairs); the sentence could say "titles are stored in NFC when they are written".
NOTE 6: tree order of a legacy pair
The link goes to the first page in the server's order position, lower(title), id (store.go:75),
flat over all levels, not the visual depth-first order; the client keeps that order in wiki.live
(wikiTree.ts:100). Deterministic and documented ("position, then title"). Whether every client-side
update of the page list (live events after an edit) keeps server order was not checked; if it does not,
the target of a legacy pair link can change after an edit until reload. Affects only pre-fix pairs.
Gates
- No allowlist, lint, size or test-skip change in the commit. File sizes: title_key.go 66,
store_edit.go 301, store_write.go 254, wikiTree.ts 106 lines.
make complexity-checkpasses. go vetwith and without-tags=integrationon internal/wiki and internal/server: clean. internal/repo tests: ok.node scripts/check-english-only.mjs: ok. ESLint on the five touched web files: clean.- No new dependency (golang.org/x/text was already direct). No public API break: doc/api-stability.md does not cover wiki title semantics; responses now carry NFC titles and the 409 rule changed for U+0130, both documented in doc/wiki.md.
- Commit type
fix, no!, no attribution line. English only.
What I ran
- Exhaustive key comparison (all code points, 300,000 random strings) Go 1.27.4 vs Node 24.12.0 and vs Node 26.13.1 (official build, checksum verified, kept outside the repository).
- The tests above with and without the fix; vitest on both Node versions; vector regeneration on both.
- Own throwaway integration tests (removed): concurrent creates, legacy NFD edit and restore, NFC growth, undelete onto a held key, rename onto a pair's key, search with NFD and NFC queries.
- Benchmarks: Go
titleTakenloop; clientresolveWikiLinkbefore and after, and a map-based variant.
What I did not check
- Real browsers (Chrome, Safari, Firefox) and their ICU versions; the browser analysis above is reasoned from the Node results, not run.
- The HTTP route end to end (store layer only, like the phase 2 reports); vue-tsc over the project.
- Postgres collations other than the test image's default (affects only
lower(title)in the tree order). - Whether client-side live updates of the page list keep the server's order (NOTE 6).
Second round: commit 7c9359374 (the delta on top of 200b1284c)
"fix: resolve wiki links against keys computed once per page list, and compose search queries". Reviewed on its own against 200b1284c, in a fresh temporary worktree (removed).
Verdict for the delta: MERGE AFTER FIXES
SHOULD-FIX 1 is closed. One small new SHOULD-FIX: normalising the search query loses matches that a decomposed query used to find in decomposed stored text.
SHOULD-FIX 1 (round 1): closed
wikiLinkResolver (web/src/wiki/lib/wikiTree.ts:70-89) builds a key-to-first-slug Map and an address
Set once per page list. wikiLinkRule (wikiLinks.ts:12) gets the resolver once per rule. I repeated the
round 1 measurement (same 500 pages, same machine, the new resolver's logic copied into the benchmark):
| Text | 200b1284c | 7c9359374, same list (cache hit) | 7c9359374, new list on each render |
|---|---|---|---|
| Index page linking all 500 pages | 146 ms | 0.6 ms | 1.1 ms |
| 1,000 links to missing pages | 520 ms | 0.2 ms | 0.8 ms |
| 10,000 links to missing pages | 5,100 ms | 1.6 ms | 2.6 ms |
The new test web/src/wiki/lib/wikiLinkCost.test.ts counts wikiTitleKey calls (500 + 2,000 on the
first render, 2,000 on the second). With wikiTree.ts and wikiLinks.ts set back to 200b1284c it fails:
"expected to be called 2500 times, but got 627750 times". It runs as a normal test.
Cache keyed on list identity
- The cache is a
WeakMap(wikiTree.ts:59), so the resolver of an old list goes when the list does. - Every renderer passes
wiki.live(WikiPageView.vue:102, WikiEditor.vue:119, WikiHistory.vue:70).liveis acomputedfilter over the store'spages(wikiTree.tswikiViews), so it is a new array wheneverpageschanges. - The store (web/src/stores/wiki.ts) never changes
pagesin place.loadTreeassigns the fetched list (line 48).applyPageassignswithPage(...), which builds a new array with a new summary object (lines 25-27, 88).resetassigns[]. A grep of web/src outside tests finds nopush,splice,sort,reverse,unshift, index assignment or field assignment (.title =,.slug =,.deleted_at =) onwiki.pagesorwiki.live. Every other consumer only reads (filter,find,breadcrumbs,buildTree). - One reactivity question: on a cache hit, MarkdownContent's computed no longer reads each page's title, so it does not track titles. That is safe only because page objects are replaced, never changed; the comment at wikiTree.ts:66-67 says so.
- Own throwaway test, with the real Pinia store and a mounted MarkdownContent bound to
wiki.live:- The links
[[Raid guide]] [[New name]] [[Fresh]] [[raid]]render first as raid / missing / missing / raid. - After
applyPagerenames page 1: missing / raid / missing / raid. - After
applyPagewithdeleted_at: all missing. - After
loadTreereturns a new list: raid / missing / fresh / raid. wiki.liveis the same array on two reads in a row.- No stale resolver at any step. The same test shows that the cache cannot see an in-place
spliceon a reactive array, which only matters if a future change mutates the list. A one-line comment in stores/wiki.ts would protect that contract (NOTE).
- The links
Tie-breaking still holds
- First in list order wins.
byKeyis set only when the key is not there yet (wikiTree.ts:77). Own test: an NFC/NFD pair resolves to the first in either order. The existing wikiTree.test.ts case also passes. - A title match wins over an address. The title map is checked before the address set
(wikiTree.ts:84). Own test: page A has address "b", page B has title "b";
[[B]]resolves to page B (by title), and[[a]]resolves by address. - Deleted pages are skipped, as before (wikiTree.ts:75).
resolveWikiLinkis kept as a thin wrapper.
Search normalisation
internal/wiki/store_search.go:25 now runs norm.NFC.String on the trimmed query.
ASCII and NFC queries: results unchanged. NFC leaves a string that is already NFC (and all ASCII) unchanged. Own before/after run: "ascii" and "ASCII page" give 1 hit each in both versions; an NFC query gives the same results in both.
SHOULD-FIX (new): a decomposed query no longer finds decomposed stored text. Page text is not normalised, and neither is an old page's title until that page is saved again. Own integration test run at 200b1284c and at 7c9359374:
Query Stored text 200b1284c 7c9359374 "Café menu" (NFD) page text with "Café menu.pdf" (NFD, for example a pasted macOS file name) 1 hit 0 hits "Résumé" (NFD) old page titled "Résumé old" (NFD, not yet saved again) 1 hit 0 hits The gain (an NFD query finds an NFC title, covered by the new assertion in title_key_integration_test.go:29-32) comes with these losses. It is rare: only pasted decomposed text is affected, and keyboards produce NFC. Fix: when
NFC(q) != q, match either form, for exampletitle ILIKE ANY($1) OR content ILIKE ANY($1)with both patterns, also in the ORDER BY. This also makes an NFC query find decomposed text, which neither version does today.MinSearchRunes(3) is checked after NFC, so a 3-code-point decomposed query such as "éx" (2 after NFC) now returns nothing. It is shorter than 3 visible characters, so this is consistent (NOTE).doc/wiki.md: "a search query is composed the same way" is accurate. It does not say that page text is not normalised; worth a clause once the fix above is in.
Gates and tests run (delta)
go test ./internal/wiki/andgo test -tags=integration ./internal/wiki/pass.go vetwith and without the integration tag: clean.- vitest: src/wiki, src/stores/wiki.test.ts and src/shared/components/MarkdownContent.wiki.test.ts, 25 files, 263 passed, 2 skipped.
- ESLint on the three touched web files: clean.
scripts/check-english-only.mjs: ok.make complexity-check: no findings. - No allowlist, limit or skip change. Commit type
fix.
Not checked (delta)
Real browsers; the end-to-end HTTP search route; vue-tsc over the project.
Third round: commit 11b7870da (search matches both forms)
Reviewed on its own against 7c9359374, in a fresh temporary worktree (removed).
Verdict for the delta: MERGE
The round 2 SHOULD-FIX is closed and no new defect was found. Notes only.
SHOULD-FIX (round 2): closed
queryForms (internal/wiki/store_search.go) returns the query composed (NFC) and, when they differ,
also decomposed (NFD). Both forms are escaped and matched with ILIKE ANY($1::text[]) on the title and
on the text.
- The new TestSearchFindsBothUnicodeForms covers all four combinations: a title in NFC, a title in NFD, text in NFC and text in NFD, each searched with an NFC query, an NFD query and an upper-case NFD query. Every query finds all 4 pages, titles first.
- With store_search.go set back to 7c9359374, both the new test and the changed assertion in TestPagesThatAlreadyShareATitleKeyStayEditable fail: each query finds only the 2 NFC pages.
- My round 2 cases now pass. "Café menu" (NFD) against NFD text, and "Résumé" against an old NFD title, find their pages. "é 50%" in either form finds the NFC title and the NFD text.
LIKE escaping without ESCAPE
The ESCAPE clause is gone. LIKE and ILIKE use the backslash as their escape character by default, and
that is the character likePattern uses. standard_conforming_strings (on in the test image) only
affects how SQL string literals are parsed. The patterns are bound parameters, not literals, so it has
no effect here.
Own test against real Postgres, with titles "Sure 100% yes", "Sure 1000 yes", "Key a_b here",
"Key axb here", Path back\slash and "Path backslash":
| Query | Result |
|---|---|
| "0% y", "100%" | only the % page |
| "a_b" | only "a_b" ("axb" not matched) |
| "x_b" | nothing |
k\s | only the backslash page |
k\\s | nothing |
\%, "%%%", "___" | nothing (a wildcard matches nothing but itself) |
Same behaviour as with ESCAPE '\'.
Query plan at the 500-page bound
Own EXPLAIN (ANALYZE, BUFFERS) on 508 live pages, each text about 39 KB, Postgres from the test image:
| Query | Plan | Time |
|---|---|---|
Before: one pattern, ESCAPE '\' | Seq Scan | 96 ms |
| After: one pattern (ASCII or NFC = NFD) | Seq Scan | 105 ms (noise) |
After: two patterns (résumé) | Seq Scan | 229 ms |
- No index was used before either: the
ORacross title and text gives a sequential scan at this size. The two trigram GIN indexes are there (migration 0089 lines 31-32) but are not chosen. - The two-pattern case costs about twice as much, as expected. It happens only for queries holding a character that decomposes, and stays linear and bounded by the page limit.
- NOTE: a GIN trigram index cannot serve
ILIKE ANY(array), so even on a much larger table the indexes can no longer help this query. At the 500-page bound that does not matter. If it ever does, usetitle ILIKE $1 OR title ILIKE $2 ...with separate parameters.
Snippet window
formIn picks the form the text actually holds (comparing lower case) and passes it to snippet, so
the window centres on the match in either form. Own test: NFD text "Café 50% off" searched in NFC
gives the snippet "Café 50% off". The new test checks the snippet for both text forms. snippet itself
is unchanged.
ASCII queries
Unchanged. An ASCII query is its own NFC and NFD, so there is one pattern, escaped the same way, with the same results ("ascii", "ASCII page"). An ASCII query such as "cafe" still finds NFD text "Café", because the base letters are there; that was also true before.
Notes
- Text stored in a mixed form is still not matched by a query spanning the mixed part, for example "Résumé" (one letter decomposed, one composed). It needs pasted half-normalised text. Edge case, not worth more patterns.
MinSearchRunesis checked on the NFC form, as in round 2.- doc/wiki.md now says page text is stored as typed and the search matches both forms. Accurate.
Gates and tests run (delta)
- The wiki store package passes, with and without the integration tag. The integration tests of internal/wiki and internal/server matching Wiki|Search|Title|Page pass.
go vetwith and without the integration tag: clean.make complexity-check: no findings. English-only check: ok.- store_search.go is 94 lines. No allowlist, limit or skip change. Commit type
fix.
Not checked (delta)
The HTTP route end to end (handler_read.go:163 passes the raw q through unchanged); Postgres
collations other than the test image's.