Fix review: H-L3-09 (wiki titles: one Unicode-aware key on the server and in the app)

From mnema-talk v3.3.13, rendered on this site.

Reviewer: fix-l3-09 review (phase 3), read-only. Fix: branch msch128/fix-h-l3-09, commit 200b1284c ("fix: wiki titles compare by one Unicode-aware key on the server and in the app"), reviewed on its own against its parent. One temporary worktree, removed when done; nothing pushed, nothing changed in the repository.

Verdict: MERGE AFTER FIXES

One SHOULD-FIX (a performance regression on the page view and the editor preview), no blocker. The fix removes the cause the finding and the refuter describe (two rules, no normalisation), for both variants (NFC/NFD look-alikes, the U+0130 disagreement), and the two key functions agree on every code point and on 300,000 random multi-character strings under the Node version CI uses.

Cause removed?

Yes. The server's titleTaken (internal/wiki/title_key.go:50) and the client's resolveWikiLink (web/src/wiki/lib/wikiTree.ts:62) both compare NFC(simpleLower each code point(NFD(title))); titles are stored NFC (cleanTitle, title_key.go:38, used by cleanText and by RestoreRevision). Every writer of wiki_pages.title goes through checkTitle (Create, Change via save, RestoreRevision via save, Undelete); there is no other writer in internal/ (grep for INSERT INTO wiki_pages and UPDATE wiki_pages SET ... title).

Do the two key functions agree? (own exhaustive comparison)

Own harness (outside the repository): the Go function copied byte for byte (same golang.org/x/text v0.43.0, Go 1.27.4: unicode.Version 17.0.0, norm.Version 17.0.0) and the TypeScript function copied into a plain Node script. Each prints the key of every code point U+0000..U+10FFFF except surrogates (1,112,064 inputs), and of 300,000 random strings of 1 to 8 code points (3/4 drawn from a pool of special-casing letters, final and medial sigma, U+0130/U+0131, titlecase digraphs, Kelvin/Angstrom/Ohm signs and other canonical singletons, composition exclusions such as U+0958, U+FB2A, U+2ADC, U+1D15E, combining marks of many combining classes in random order, Hangul jamo L/V/T and syllables, astral cased scripts, the Unicode 16 and 17 additions; 1/4 uniform over all planes).

ComparisonSingle code points that differMulti-character strings that differ
Go 1.27.4 vs Node 26.13.1 (ICU 78.3, Unicode 17; the version every CI job uses)00 of 300,000
Go 1.27.4 vs Node 24.12.0 (ICU 77.1, Unicode 16)2815,742 of 300,000, every one containing one of the 28

The 28 are exactly the case pairs new in Unicode 17: U+A7CE, U+A7D2, U+A7D4 (Latin Extended-D) and U+16EA0..U+16EB8 (Beria Erfe capitals). Go maps them to their lower case, Node 24 leaves them. No normalisation difference at all (every difference is a case mapping). This matches the author's claim. All 28 lie above U+A000, so the vector sweep (U+0020..U+9FFF) does not contain them.

What happens in production when server and browser disagree

The server (Go, Unicode 17) is the only place that enforces uniqueness; the browser only resolves links, with whatever ICU the reader's browser ships.

  • Browser older than the server (the 28 characters today, in any browser below Unicode 17): the server folds more than the browser. A page "X" whose title holds a new capital, linked as [[x]] with the small letter, shows as a missing link (dotted underline, web/src/wiki/lib/wikiLinks.ts:18) for that reader; creating it is refused by the server as a taken title. No duplicate can arise: two titles the browser keeps apart but the server joins cannot both be created.
  • Browser newer than the server (a future Unicode in the browser before Go updates): the browser joins two titles the server keeps apart. Both pages exist (they differ only in the case of a new letter, so they look different); a link resolves to the first in tree order, as for an old shared pair. No duplicate in the server's sense.
  • A Go upgrade that adds case pairs turns two existing distinct pages into a shared-key pair: handled exactly like the pre-fix pairs (both stay, both editable, save compares keys).
  • Nothing worse: no error path, no crash, no permission effect (a link only navigates among pages the reader already sees). Worst case is a link shown as missing for 28 rare characters in older browsers.

Recorded tests: normal tests, fail without the fix

  • wikiTitleFold.test.ts: it.fails is now a normal it; the known-disagreement list is now [].
  • Revert run (store_write.go, store_edit.go, wikiTree.ts back to the parent; title_key.go reduced to titleKey plus the old cleanTitle so the tests compile):
    • go test ./internal/wiki/: TestATitleIsStoredComposed fails (3 cases).
    • go test -tags=integration -run TestPagesThatAlreadyShareATitleKeyStayEditable ./internal/wiki/: fails.
    • go test -tags=integration -run TestWikiTitlePairsMatchTheServer ./internal/server: 9 mismatches (the 8 NFC/NFD pairs and the U+0130 pair), as the commit message says.
    • vitest src/wiki/lib/: 2 fail (the disagreement test and the new resolveWikiLink test).
    • TestTitleKeysMatchTheVectors cannot fail on a revert (it tests titleKey itself, which the revert keeps); it guards drift between the two implementations, not the store.
  • With the fix: go test ./internal/wiki/ ok; go test -tags=integration ./internal/wiki/ ok (whole package); go test -tags=integration -run 'TestWiki' ./internal/server ok; vitest src/wiki/lib/ 12 files, 116 passed, 2 skipped, on Node 24.12.0 and on Node 26.13.1.

The specific questions

İstanbul next to Istanbul. TestTitlesAreOneLineAndUniqueIgnoringCase (internal/wiki/store_integration_test.go:517) is unchanged by the commit and passes, with and without the fix. It does not contain U+0130. Its "a lookalike is the same title" (line 524) is about a zero-width space, which renders as nothing; U+0130 renders with a visible dot, so allowing "İstanbul" next to "Istanbul" does not contradict its intent. The change is documented (doc/wiki.md) and makes the server agree with what the client resolves, which is the property. Accepted as a deliberate behaviour change. (Side effect worth knowing, not a defect under the documented rule: [[istanbul]] does not resolve to "İstanbul"; the rule is locale-independent.)

Race. Not widened, and there is none. Every write path (Create store_write.go:150, inTx used by Undelete store_write.go:210, Change store_edit.go:96, RestoreRevision store_edit.go:260) takes pg_advisory_xact_lock(hashtext('mnema.wiki')) (store.go:150) before the check and holds it until commit, so check and insert/update are serialised. That was true before (the old SQL EXISTS ran under the same lock) and is true now. Own run: 20 rounds of 4 concurrent Create calls whose titles share one key (NFC, NFD, upper and lower case): exactly one succeeded per round, the others got errTitleTaken.

The 500-title bound. MaxPages = 500 (internal/wiki/model.go:23) is enforced by checkCount (store_write.go:126) in Create and Undelete, under the tree lock; deleted pages are excluded from both the count and titleTaken's query. Rows written before the limit could exceed it, but the cost is linear in any case. Cost measured (own Go benchmark, 500 titles of ~35 characters plus a 100-character title): 0.70 ms and ~740 KB allocated per check, plus reading up to 500 short rows. It runs only on create, title change, restore and undelete (rate-limited routes), inside the lock. Acceptable.

Stored NFC on an unrelated edit. The web editor always sends the title (web/src/wiki/composables/useWikiEditor.ts:29), so the first edit of a page stored in NFD rewrites its title bytes to NFC. Checked with an own integration test: a text-only edit of an NFD-titled page writes a revision with the NFC title. Effects: the slug does not change (slugs come from the title only on create; rename takes an explicit slug); links resolve by key, so no link breaks; history gets a revision whose title differs only in bytes (see NOTE 3); search is ILIKE on the stored bytes (see NOTE 4).

Restore of an old revision. Goes through cleanTitle (store_edit.go:272) and save, so the title is stored NFC and checked by key. Own test: restore onto a key held by another page is refused (also in the author's test). See NOTE 3 for the no-op restore and NOTE 2 for titles that grow under NFC.

Rename onto a key an existing pair holds. Own test: two legacy pages "Xé" (NFC) and "Xé" (NFD), a third page renamed to "XÉ" is refused (errTitleTaken). One member of the pair renamed to another case of its own key ("XÉ") is allowed, which creates no new duplicate. Undelete of a legacy page onto a key now held by another page is refused.

The vector file. internal/wiki/testdata/title_keys.json is pure ASCII (no byte outside 0x20..0x7E and newline): the writer escapes every UTF-16 unit, so astral characters are valid JSON surrogate pairs, which Go's encoding/json decodes to the right code point (the Go test passes on them). Regenerated with WIKI_KEYS_WRITE=1 under Node 24.12.0 and under Node 26.13.1: both outputs are byte for byte identical to the committed file (46 vectors). Stable across these versions because the sweep stops below U+A000, where Unicode 16 and 17 have the same case data. A future Node whose ICU adds a case pair below U+A000 would make expect(v.map(x => x.title)).toEqual(titles()) fail until the file is regenerated; that is the intended alarm, not a flake.

Findings

web/src/wiki/lib/wikiTree.ts:62-69. resolveWikiLink computes wikiTitleKey(p.title) (two normalisations and a per-code-point loop) for every live page, for every [[...]] in the text, on every render. wikiLinkRule (wikiLinks.ts:11-23) calls it per match, and MarkdownContent recomputes the whole render whenever content or wiki.live changes (shared/components/MarkdownContent.vue:51-56); the editor's side-by-side preview (WikiEditor.vue:119) re-renders on each keystroke.

Measured in Node 24 on an Apple-silicon Mac (own benchmark, 500 pages of ~35 characters):

TextBeforeAfter
An index page linking all 500 pages2.5 ms146 ms
1,000 links to missing pages15 ms520 ms
10,000 links to missing pages (fits in the 50 KB content limit)139 ms5,100 ms

So on a full wiki, typing in the editor of an index page costs ~150 ms per keystroke on a fast laptop (several times that on a phone), and a page full of [[x]] freezes every reader's tab for seconds. Fix: compute each page's key once per page list, for example a Map from key to the first slug in list order built once in wikiLinkRule(pages) (or memoised in wikiViews), and look links up in it. Measured: 1.4 ms per render for the 500-link index page, including building the map. The "first in tree order" rule is kept by inserting only when the key is not yet in the map.

NOTE 2: titles that grow under NFC are refused at a length the writer did not type

title_key.go:43 checks the length again after NFC (needed: the column is VARCHAR(100), Postgres counts code points). For composition-excluded characters NFC gives two code points for one (U+0958..U+095F, U+09DC.., U+FB1D, U+FB2A..U+FB4E, U+2ADC, U+1D15E..). Own test: creating a title of 60 × U+0958 is refused "title must be at most 100 characters" (before: accepted). A legacy page whose stored title is 60 × U+0958 can no longer be edited from the web app (which always sends the title) until the title is retyped shorter or in decomposed form; a text-only PATCH still works. Restoring an old revision with such a title is refused the same way. Keyboards produce these letters decomposed, so this needs pasted precomposed forms, many of them, near the limit; very unlikely. Possible remedy: in save, keep the current bytes when the key is unchanged and the NFC form would not fit. Not required for merge.

NOTE 3: a title-only normalisation writes a revision, and a no-op restore is no longer refused

reviseText (store_edit.go:140) and restorable (store_edit.go:297) compare titles by bytes. Own test, legacy NFD page: an edit with unchanged visible title and text writes revision 2 (title bytes NFC); a restore of revision 1 (NFD, same text) is then accepted and writes revision 3 that changes nothing visible, where errNothingNew would be the expected answer. Harmless (history noise, once per legacy page); comparing cleanTitle(target.Title) in restorable would restore the refusal.

NOTE 4: wiki search does not normalise the query

internal/wiki/store_search.go:29 matches title ILIKE on stored bytes. Titles are now NFC, so a query typed in NFD finds no title (own test: NFD query 0 hits, NFC query 1 hit). Before the fix the same gap existed in the other direction for NFD titles, and content is not normalised either, so this is not a regression of the fix, but normalising the query to NFC would complete it.

NOTE 5: "Titles are stored in Unicode NFC" holds for pages written after the fix

doc/wiki.md: a legacy NFD page keeps its bytes until it is saved (Undelete does not normalise, store_write.go:187-194, own test). The following paragraph on old pages covers the consequence (pairs); the sentence could say "titles are stored in NFC when they are written".

NOTE 6: tree order of a legacy pair

The link goes to the first page in the server's order position, lower(title), id (store.go:75), flat over all levels, not the visual depth-first order; the client keeps that order in wiki.live (wikiTree.ts:100). Deterministic and documented ("position, then title"). Whether every client-side update of the page list (live events after an edit) keeps server order was not checked; if it does not, the target of a legacy pair link can change after an edit until reload. Affects only pre-fix pairs.

Gates

  • No allowlist, lint, size or test-skip change in the commit. File sizes: title_key.go 66, store_edit.go 301, store_write.go 254, wikiTree.ts 106 lines. make complexity-check passes.
  • go vet with and without -tags=integration on internal/wiki and internal/server: clean. internal/repo tests: ok. node scripts/check-english-only.mjs: ok. ESLint on the five touched web files: clean.
  • No new dependency (golang.org/x/text was already direct). No public API break: doc/api-stability.md does not cover wiki title semantics; responses now carry NFC titles and the 409 rule changed for U+0130, both documented in doc/wiki.md.
  • Commit type fix, no !, no attribution line. English only.

What I ran

  • Exhaustive key comparison (all code points, 300,000 random strings) Go 1.27.4 vs Node 24.12.0 and vs Node 26.13.1 (official build, checksum verified, kept outside the repository).
  • The tests above with and without the fix; vitest on both Node versions; vector regeneration on both.
  • Own throwaway integration tests (removed): concurrent creates, legacy NFD edit and restore, NFC growth, undelete onto a held key, rename onto a pair's key, search with NFD and NFC queries.
  • Benchmarks: Go titleTaken loop; client resolveWikiLink before and after, and a map-based variant.

What I did not check

  • Real browsers (Chrome, Safari, Firefox) and their ICU versions; the browser analysis above is reasoned from the Node results, not run.
  • The HTTP route end to end (store layer only, like the phase 2 reports); vue-tsc over the project.
  • Postgres collations other than the test image's default (affects only lower(title) in the tree order).
  • Whether client-side live updates of the page list keep the server's order (NOTE 6).

Second round: commit 7c9359374 (the delta on top of 200b1284c)

"fix: resolve wiki links against keys computed once per page list, and compose search queries". Reviewed on its own against 200b1284c, in a fresh temporary worktree (removed).

Verdict for the delta: MERGE AFTER FIXES

SHOULD-FIX 1 is closed. One small new SHOULD-FIX: normalising the search query loses matches that a decomposed query used to find in decomposed stored text.

SHOULD-FIX 1 (round 1): closed

wikiLinkResolver (web/src/wiki/lib/wikiTree.ts:70-89) builds a key-to-first-slug Map and an address Set once per page list. wikiLinkRule (wikiLinks.ts:12) gets the resolver once per rule. I repeated the round 1 measurement (same 500 pages, same machine, the new resolver's logic copied into the benchmark):

Text200b1284c7c9359374, same list (cache hit)7c9359374, new list on each render
Index page linking all 500 pages146 ms0.6 ms1.1 ms
1,000 links to missing pages520 ms0.2 ms0.8 ms
10,000 links to missing pages5,100 ms1.6 ms2.6 ms

The new test web/src/wiki/lib/wikiLinkCost.test.ts counts wikiTitleKey calls (500 + 2,000 on the first render, 2,000 on the second). With wikiTree.ts and wikiLinks.ts set back to 200b1284c it fails: "expected to be called 2500 times, but got 627750 times". It runs as a normal test.

Cache keyed on list identity

  • The cache is a WeakMap (wikiTree.ts:59), so the resolver of an old list goes when the list does.
  • Every renderer passes wiki.live (WikiPageView.vue:102, WikiEditor.vue:119, WikiHistory.vue:70). live is a computed filter over the store's pages (wikiTree.ts wikiViews), so it is a new array whenever pages changes.
  • The store (web/src/stores/wiki.ts) never changes pages in place. loadTree assigns the fetched list (line 48). applyPage assigns withPage(...), which builds a new array with a new summary object (lines 25-27, 88). reset assigns []. A grep of web/src outside tests finds no push, splice, sort, reverse, unshift, index assignment or field assignment (.title =, .slug =, .deleted_at =) on wiki.pages or wiki.live. Every other consumer only reads (filter, find, breadcrumbs, buildTree).
  • One reactivity question: on a cache hit, MarkdownContent's computed no longer reads each page's title, so it does not track titles. That is safe only because page objects are replaced, never changed; the comment at wikiTree.ts:66-67 says so.
  • Own throwaway test, with the real Pinia store and a mounted MarkdownContent bound to wiki.live:
    • The links [[Raid guide]] [[New name]] [[Fresh]] [[raid]] render first as raid / missing / missing / raid.
    • After applyPage renames page 1: missing / raid / missing / raid.
    • After applyPage with deleted_at: all missing.
    • After loadTree returns a new list: raid / missing / fresh / raid.
    • wiki.live is the same array on two reads in a row.
    • No stale resolver at any step. The same test shows that the cache cannot see an in-place splice on a reactive array, which only matters if a future change mutates the list. A one-line comment in stores/wiki.ts would protect that contract (NOTE).

Tie-breaking still holds

  • First in list order wins. byKey is set only when the key is not there yet (wikiTree.ts:77). Own test: an NFC/NFD pair resolves to the first in either order. The existing wikiTree.test.ts case also passes.
  • A title match wins over an address. The title map is checked before the address set (wikiTree.ts:84). Own test: page A has address "b", page B has title "b"; [[B]] resolves to page B (by title), and [[a]] resolves by address.
  • Deleted pages are skipped, as before (wikiTree.ts:75). resolveWikiLink is kept as a thin wrapper.

Search normalisation

internal/wiki/store_search.go:25 now runs norm.NFC.String on the trimmed query.

  • ASCII and NFC queries: results unchanged. NFC leaves a string that is already NFC (and all ASCII) unchanged. Own before/after run: "ascii" and "ASCII page" give 1 hit each in both versions; an NFC query gives the same results in both.

  • SHOULD-FIX (new): a decomposed query no longer finds decomposed stored text. Page text is not normalised, and neither is an old page's title until that page is saved again. Own integration test run at 200b1284c and at 7c9359374:

    QueryStored text200b1284c7c9359374
    "Café menu" (NFD)page text with "Café menu.pdf" (NFD, for example a pasted macOS file name)1 hit0 hits
    "Résumé" (NFD)old page titled "Résumé old" (NFD, not yet saved again)1 hit0 hits

    The gain (an NFD query finds an NFC title, covered by the new assertion in title_key_integration_test.go:29-32) comes with these losses. It is rare: only pasted decomposed text is affected, and keyboards produce NFC. Fix: when NFC(q) != q, match either form, for example title ILIKE ANY($1) OR content ILIKE ANY($1) with both patterns, also in the ORDER BY. This also makes an NFC query find decomposed text, which neither version does today.

  • MinSearchRunes (3) is checked after NFC, so a 3-code-point decomposed query such as "éx" (2 after NFC) now returns nothing. It is shorter than 3 visible characters, so this is consistent (NOTE).

  • doc/wiki.md: "a search query is composed the same way" is accurate. It does not say that page text is not normalised; worth a clause once the fix above is in.

Gates and tests run (delta)

  • go test ./internal/wiki/ and go test -tags=integration ./internal/wiki/ pass.
  • go vet with and without the integration tag: clean.
  • vitest: src/wiki, src/stores/wiki.test.ts and src/shared/components/MarkdownContent.wiki.test.ts, 25 files, 263 passed, 2 skipped.
  • ESLint on the three touched web files: clean. scripts/check-english-only.mjs: ok. make complexity-check: no findings.
  • No allowlist, limit or skip change. Commit type fix.

Not checked (delta)

Real browsers; the end-to-end HTTP search route; vue-tsc over the project.

Third round: commit 11b7870da (search matches both forms)

Reviewed on its own against 7c9359374, in a fresh temporary worktree (removed).

Verdict for the delta: MERGE

The round 2 SHOULD-FIX is closed and no new defect was found. Notes only.

SHOULD-FIX (round 2): closed

queryForms (internal/wiki/store_search.go) returns the query composed (NFC) and, when they differ, also decomposed (NFD). Both forms are escaped and matched with ILIKE ANY($1::text[]) on the title and on the text.

  • The new TestSearchFindsBothUnicodeForms covers all four combinations: a title in NFC, a title in NFD, text in NFC and text in NFD, each searched with an NFC query, an NFD query and an upper-case NFD query. Every query finds all 4 pages, titles first.
  • With store_search.go set back to 7c9359374, both the new test and the changed assertion in TestPagesThatAlreadyShareATitleKeyStayEditable fail: each query finds only the 2 NFC pages.
  • My round 2 cases now pass. "Café menu" (NFD) against NFD text, and "Résumé" against an old NFD title, find their pages. "é 50%" in either form finds the NFC title and the NFD text.

LIKE escaping without ESCAPE

The ESCAPE clause is gone. LIKE and ILIKE use the backslash as their escape character by default, and that is the character likePattern uses. standard_conforming_strings (on in the test image) only affects how SQL string literals are parsed. The patterns are bound parameters, not literals, so it has no effect here.

Own test against real Postgres, with titles "Sure 100% yes", "Sure 1000 yes", "Key a_b here", "Key axb here", Path back\slash and "Path backslash":

QueryResult
"0% y", "100%"only the % page
"a_b"only "a_b" ("axb" not matched)
"x_b"nothing
k\sonly the backslash page
k\\snothing
\%, "%%%", "___"nothing (a wildcard matches nothing but itself)

Same behaviour as with ESCAPE '\'.

Query plan at the 500-page bound

Own EXPLAIN (ANALYZE, BUFFERS) on 508 live pages, each text about 39 KB, Postgres from the test image:

QueryPlanTime
Before: one pattern, ESCAPE '\'Seq Scan96 ms
After: one pattern (ASCII or NFC = NFD)Seq Scan105 ms (noise)
After: two patterns (résumé)Seq Scan229 ms
  • No index was used before either: the OR across title and text gives a sequential scan at this size. The two trigram GIN indexes are there (migration 0089 lines 31-32) but are not chosen.
  • The two-pattern case costs about twice as much, as expected. It happens only for queries holding a character that decomposes, and stays linear and bounded by the page limit.
  • NOTE: a GIN trigram index cannot serve ILIKE ANY(array), so even on a much larger table the indexes can no longer help this query. At the 500-page bound that does not matter. If it ever does, use title ILIKE $1 OR title ILIKE $2 ... with separate parameters.

Snippet window

formIn picks the form the text actually holds (comparing lower case) and passes it to snippet, so the window centres on the match in either form. Own test: NFD text "Café 50% off" searched in NFC gives the snippet "Café 50% off". The new test checks the snippet for both text forms. snippet itself is unchanged.

ASCII queries

Unchanged. An ASCII query is its own NFC and NFD, so there is one pattern, escaped the same way, with the same results ("ascii", "ASCII page"). An ASCII query such as "cafe" still finds NFD text "Café", because the base letters are there; that was also true before.

Notes

  • Text stored in a mixed form is still not matched by a query spanning the mixed part, for example "Résumé" (one letter decomposed, one composed). It needs pasted half-normalised text. Edge case, not worth more patterns.
  • MinSearchRunes is checked on the NFC form, as in round 2.
  • doc/wiki.md now says page text is stored as typed and the search matches both forms. Accurate.

Gates and tests run (delta)

  • The wiki store package passes, with and without the integration tag. The integration tests of internal/wiki and internal/server matching Wiki|Search|Title|Page pass.
  • go vet with and without the integration tag: clean. make complexity-check: no findings. English-only check: ok.
  • store_search.go is 94 lines. No allowlist, limit or skip change. Commit type fix.

Not checked (delta)

The HTTP route end to end (handler_read.go:163 passes the raw q through unchanged); Postgres collations other than the test image's.

Something wrong on this page? Tell us.