feat(editor): Shiki syntax highlighting in the file editor (CodeMirror)

Bring the CodeMirror file editor up to the same rich highlighting as the Shiki
file view, so toggling edit <-> view is visually consistent. lezer collapses far
more tokens than TextMate (import/from/const are all "keyword"), so a theme
remap can't reach parity — instead, project real Shiki tokens onto decorations.

- Worker: add highlightTokens — tokenize with an arbitrary registered TextMate
  theme and return per-line styled runs with offsets. The theme object ships to
  the worker once per name; later calls send only the name.
- New shikiHighlight CodeMirror extension: a StateField of mark decorations
  built from worker tokens. Re-tokenizes on a short idle (off the keystroke
  path) and maps decorations through edits so colors persist while typing.
- flexokiTheme: add { syntaxColors: false } to keep only the editor UI theme,
  so the lezer highlighter doesn't compete with the Shiki decorations.
- FilesView: enable Shiki highlighting (same language resolver as the file view
  → identical language) and drop lezer token colors when it's active. lezer
  language stays on for indentation/folding/brackets.
This commit is contained in:
Bohdan Triapitsyn
2026-06-16 01:08:54 +03:00
parent c22e1cbb15
commit 782982cdd3
6 changed files with 232 additions and 5 deletions
@@ -1,5 +1,5 @@
import MarkdownShikiWorkerUrl from './markdown-shiki.worker.ts?worker&url';
import type { MarkdownWorkerRequest, MarkdownWorkerResponse } from './markdown-worker-protocol';
import type { MarkdownTokenRun, MarkdownWorkerRequest, MarkdownWorkerResponse } from './markdown-worker-protocol';
// Main-thread client for the markdown Shiki worker. Moves syntax tokenization
// off the UI thread: a closed code block is shipped to the worker, which returns
@@ -12,10 +12,14 @@ type PendingResolver = (response: MarkdownWorkerResponse | null) => void;
let worker: Worker | undefined;
let nextId = 0;
const pending = new Map<number, PendingResolver>();
// Theme names whose full definition we've already shipped to the live worker, so
// repeat tokenization sends only the name (not the whole theme object) again.
const sentThemes = new Set<string>();
const failAll = (): void => {
pending.forEach((resolve) => resolve(null));
pending.clear();
sentThemes.clear();
worker?.terminate();
worker = undefined;
};
@@ -68,3 +72,31 @@ export const highlightLinesInWorker = async (code: string, lang: string): Promis
const response = await request((id) => ({ type: 'highlightLines', id, code, lang }));
return response?.type === 'highlightLines' ? response.lines : null;
};
/**
* Tokenize `code` with the given resolved TextMate theme and return per-line
* styled runs with offsets — for building CodeMirror decorations that match the
* Shiki file view exactly. The full theme object is shipped only the first time
* a theme name is seen by the live worker. Resolves to `null` on failure.
*/
export const highlightTokensInWorker = async (
code: string,
lang: string,
themeName: string,
theme: unknown,
): Promise<MarkdownTokenRun[][] | null> => {
const needsTheme = !sentThemes.has(themeName);
const response = await request((id) => ({
type: 'highlightTokens',
id,
code,
lang,
themeName,
...(needsTheme ? { theme } : {}),
}));
if (response?.type === 'highlightTokens') {
sentThemes.add(themeName);
return response.lines;
}
return null;
};