Review follow-up on the #2769 highlight caches.
Fingerprint strength. The block/highlight caches are now global and
content-addressed, so a hash collision no longer mis-colors a block — it returns
a *different* block's rendered HTML and shows the user source they never wrote.
Length + one 32-bit FNV-1a is not enough key space for that failure mode at
session scale. `contentFingerprint` now combines two independent 32-bit
multiplicative hashes with a final avalanche (~64 bits); two multiplies per
character are free next to Shiki tokenization.
Streaming churn. Content addressing made every streaming step of the trailing
`live` block insert a new cache entry, so one long message evicted the settled
`full` blocks the fix exists to keep warm. `full` and `live` blocks now use
separate caches; the live cache is small (32 entries / 2MB) because it only has
to absorb repeat renders of the same step.
Redundant worker-side caches. `markdown-worker.ts` is the only sender to the
Shiki worker, and its client cache is larger than the worker-side ones, so the
worker caches could not serve a hit the client had not already served — they
only duplicated up to 48MB of payloads in a second heap. Removed; the reason
memoization belongs on the client is now documented there, along with why only
`highlightTokens` carries a theme in its key.
Dead `cacheKey` plumbing. `renderMarkdownBlocks` kept a `cacheKey` parameter it
only `void`-ed. Removed it and the now-unused `useMorphdomMarkdown` prop; the
remaining call-site local is renamed `fadeKey` for what it actually keys.
Tests: image-mode cache identity, streaming-does-not-evict-settled-blocks,
live-cache reuse, and a 20k same-length-source fingerprint collision check.
Each new guard was verified to fail without its fix.
Use content fingerprints instead of full source as cache keys, record
entry sizes once (no TextEncoder/JSON.stringify on get/evict), and
highlight multiple markdown fences concurrently.
Co-authored-by: Serhii Dziupin <makeittech@users.noreply.github.com>
Bring the CodeMirror file editor up to the same rich highlighting as the Shiki
file view, so toggling edit <-> view is visually consistent. lezer collapses far
more tokens than TextMate (import/from/const are all "keyword"), so a theme
remap can't reach parity — instead, project real Shiki tokens onto decorations.
- Worker: add highlightTokens — tokenize with an arbitrary registered TextMate
theme and return per-line styled runs with offsets. The theme object ships to
the worker once per name; later calls send only the name.
- New shikiHighlight CodeMirror extension: a StateField of mark decorations
built from worker tokens. Re-tokenizes on a short idle (off the keystroke
path) and maps decorations through edits so colors persist while typing.
- flexokiTheme: add { syntaxColors: false } to keep only the editor UI theme,
so the lezer highlighter doesn't compete with the Shiki decorations.
- FilesView: enable Shiki highlighting (same language resolver as the file view
→ identical language) and drop lezer token colors when it's active. lezer
language stays on for indentation/folding/brackets.
Route all non-markdown code highlighting through the off-main-thread Shiki
worker, removing react-syntax-highlighter and prismjs entirely.
- Extend the worker with highlightLines: tokenize a whole block once and return
per-line inner HTML, so per-line layouts (diffs, gutters, virtualization) make
one worker call instead of one highlighter per line.
- Add shared WorkerHighlightedCode (whole-block) and useWorkerHighlightedLines
(per-line) primitives. Colors resolve via the --md-syntax-* CSS variables, so
theme changes never re-highlight.
- Migrate all 12 react-syntax-highlighter call sites: PermissionCard,
ToolPart, ContextSidebarTab, ToolOutputDialog (whole block) and
DiffPreview/WritePreview (per line).
- Migrate VirtualizedCodeBlock off prismjs to the worker, keeping virtua
virtualization; whole-block tokenization also restores cross-line syntax
context that per-line highlighting lost.
- Drop react-syntax-highlighter (+types) from ui and web, prismjs (+types) from
ui, and the orphaned create-element type shim.
Tokenize closed code blocks in a dedicated Shiki Web Worker instead of calling
the shared highlighter synchronously on the UI thread. This removes the
one-shot main-thread highlight stall when a code fence closes on a large block.
Streaming behavior is unchanged: the open (streaming) fence still renders as
plain text and is highlighted once on close. On any worker failure the block
keeps its escaped plain code — highlighting never falls back onto the main
thread.
- Add markdownShikiThemeDefinition (dependency-free CSS-variable theme) so the
worker can use the theme without pulling in @pierre/diffs / React.
- Add markdown-worker-protocol, markdown-shiki.worker, and the main-thread
markdown-worker client.
- Route highlightCodeBlocks through the worker; keep the size/VSCode line guard
and mermaid skip on the main thread.
- Add shiki as a direct dependency (was transitive via @pierre/diffs).
Serhii Dziupin