Parakeet is an offline model trained on whole utterances, so re-decoding
the growing buffer to animate a live transcript cost O(n^2) work for a
result the final decode replaced. Sessions now decode once per committed
segment, and the composer shows a scrolling waveform of the mic level
instead of running text.
Long dictations split at a pause once past 60s (hard cap 90s) instead of
on a blind 15s timer, so cuts no longer land mid-word. Committed segments
decode while the user is still speaking: a 185s dictation returns 4.1s
after stop instead of 11.0s, with identical text (816 vs 817 words).
Also fixes two ways the stream manager could silently drop transcribed
audio. It now counts the commits it issued instead of trusting the
session's echoed events, so a commit still in flight when the client
finishes can no longer be left out of the final text. And segment
byte/peak accounting is reset where the commit is issued rather than when
the event arrives, which could mistake the tail of a dictation for
silence and clear it.
Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).
Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
segments auto-commit every ~15s with silence suppression and adaptive
finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
worker process — auto-download with progress, staged extraction with
verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
download/delete): Parakeet TDT v2 (English) and v3 (25 European
languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
cancel / insert / insert-and-send actions; failed transcriptions keep
their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
matching paddings/typography/gaps) — no layout shift when toggling
Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
worker via /api/dictation/tts/speak, managed by the shared model
pipeline; sentence-pipelined playback keeps time-to-first-audio at
~1 sentence regardless of message length, and stop cancels in-flight
synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
interword slashes aloud, and removes only absolute file paths
Settings:
- Voice page unified: a single read-aloud toggle owns all playback
options (the confusing "Enable Voice Mode" is gone); a new "Enable
voice input" toggle (default on, persisted to settings.json) hides
the composer mic entirely when disabled
Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
(https://localhost) was missing from the packaged-client allowlist,
403-ing every WS connection — root cause of the old mobile SSE lock,
which is now removed for all transports
Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
explicitly validates the UI session and origin, with oc_url_token
narrowly allowlisted and covered by tests; the dictation socket mints
a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
through runtimeFetch/getRuntimeUrlResolver, and runtime switches
reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
process in that runtime)
CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.
New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
Bohdan Triapitsyn