Parakeet is an offline model trained on whole utterances, so re-decoding
the growing buffer to animate a live transcript cost O(n^2) work for a
result the final decode replaced. Sessions now decode once per committed
segment, and the composer shows a scrolling waveform of the mic level
instead of running text.
Long dictations split at a pause once past 60s (hard cap 90s) instead of
on a blind 15s timer, so cuts no longer land mid-word. Committed segments
decode while the user is still speaking: a 185s dictation returns 4.1s
after stop instead of 11.0s, with identical text (816 vs 817 words).
Also fixes two ways the stream manager could silently drop transcribed
audio. It now counts the commits it issued instead of trusting the
session's echoed events, so a commit still in flight when the client
finishes can no longer be left out of the final text. And segment
byte/peak accounting is reset where the commit is issued rather than when
the event arrives, which could mistake the tail of a dictation for
silence and clear it.
The dictation overlay is absolutely positioned over the composer, so the
transcript could not expand it — long dictations clipped after two lines.
ComposerDictation now measures the transcript text block (not the flex-1
container, which would feed the composer's own height back and creep a few
px per update) and reports it to ChatInput, which feeds it into the
textarea autosize: same line cap as typing, transcript area scrolls past
it and follows the newest words. Idle/unmount releases the height, and an
idle sibling instance (mobile footer + wrapper engine) can no longer zero
the active one's report.
Mobile composer redesign: when the keyboard is closed the input collapses
into a narrow pill (sessions, attach, placeholder, mic) with a round
new-session button that fades away on the draft screen. Model and agent
selectors move into a row above the textarea; the draft project/branch
pickers and the attachment menu become searchable bottom sheets reusing
MobileOverlayPanel; a drag handle (also available while dictating) swipes
the composer into and out of a fullscreen mode.
Keyboard-lifecycle hardening: composer controls (agent cycle, dictation
and its overlay controls) no longer steal focus and dismiss the keyboard;
overlays reopen the keyboard on close via a debounced restore chain that
survives menu-to-picker handoffs and skips the native file picker; open
overlays and dictation keep the composer expanded. Dictation starts
directly from the pill and its overlay content fades in after the shape
settles. The keyboard slide compensates the pill-to-full height change in
one motion, and the mobile highlight mirror is disabled so the caret
always matches the text layout.
Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).
Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
segments auto-commit every ~15s with silence suppression and adaptive
finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
worker process — auto-download with progress, staged extraction with
verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
download/delete): Parakeet TDT v2 (English) and v3 (25 European
languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
cancel / insert / insert-and-send actions; failed transcriptions keep
their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
matching paddings/typography/gaps) — no layout shift when toggling
Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
worker via /api/dictation/tts/speak, managed by the shared model
pipeline; sentence-pipelined playback keeps time-to-first-audio at
~1 sentence regardless of message length, and stop cancels in-flight
synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
interword slashes aloud, and removes only absolute file paths
Settings:
- Voice page unified: a single read-aloud toggle owns all playback
options (the confusing "Enable Voice Mode" is gone); a new "Enable
voice input" toggle (default on, persisted to settings.json) hides
the composer mic entirely when disabled
Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
(https://localhost) was missing from the packaged-client allowlist,
403-ing every WS connection — root cause of the old mobile SSE lock,
which is now removed for all transports
Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
explicitly validates the UI session and origin, with oc_url_token
narrowly allowlisted and covered by tests; the dictation socket mints
a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
through runtimeFetch/getRuntimeUrlResolver, and runtime switches
reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
process in that runtime)
CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.
New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
Bohdan Triapitsyn