Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).
Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
segments auto-commit every ~15s with silence suppression and adaptive
finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
worker process — auto-download with progress, staged extraction with
verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
download/delete): Parakeet TDT v2 (English) and v3 (25 European
languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
cancel / insert / insert-and-send actions; failed transcriptions keep
their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
matching paddings/typography/gaps) — no layout shift when toggling
Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
worker via /api/dictation/tts/speak, managed by the shared model
pipeline; sentence-pipelined playback keeps time-to-first-audio at
~1 sentence regardless of message length, and stop cancels in-flight
synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
interword slashes aloud, and removes only absolute file paths
Settings:
- Voice page unified: a single read-aloud toggle owns all playback
options (the confusing "Enable Voice Mode" is gone); a new "Enable
voice input" toggle (default on, persisted to settings.json) hides
the composer mic entirely when disabled
Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
(https://localhost) was missing from the packaged-client allowlist,
403-ing every WS connection — root cause of the old mobile SSE lock,
which is now removed for all transports
Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
explicitly validates the UI session and origin, with oc_url_token
narrowly allowlisted and covered by tests; the dictation socket mints
a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
through runtimeFetch/getRuntimeUrlResolver, and runtime switches
reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
process in that runtime)
CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.
New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
Bohdan Triapitsyn
de1b85ac56