Files

T

Bohdan Triapitsyn de1b85ac56 feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018 )

Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).

Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
  seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
  segments auto-commit every ~15s with silence suppression and adaptive
  finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
  worker process — auto-download with progress, staged extraction with
  verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
  download/delete): Parakeet TDT v2 (English) and v3 (25 European
  languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
  cancel / insert / insert-and-send actions; failed transcriptions keep
  their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
  Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
  matching paddings/typography/gaps) — no layout shift when toggling

Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
  worker via /api/dictation/tts/speak, managed by the shared model
  pipeline; sentence-pipelined playback keeps time-to-first-audio at
  ~1 sentence regardless of message length, and stop cancels in-flight
  synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
  interword slashes aloud, and removes only absolute file paths

Settings:
- Voice page unified: a single read-aloud toggle owns all playback
  options (the confusing "Enable Voice Mode" is gone); a new "Enable
  voice input" toggle (default on, persisted to settings.json) hides
  the composer mic entirely when disabled

Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
  impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
  (https://localhost) was missing from the packaged-client allowlist,
  403-ing every WS connection — root cause of the old mobile SSE lock,
  which is now removed for all transports

Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
  explicitly validates the UI session and origin, with oc_url_token
  narrowly allowlisted and covered by tests; the dictation socket mints
  a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
  through runtimeFetch/getRuntimeUrlResolver, and runtime switches
  reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
  process in that runtime)

CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.

New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)

2026-07-04 02:48:07 +03:00

3.1 KiB

Raw Blame History

Dictation module

Server-authoritative streaming speech-to-text for the chat composer, plus local text-to-speech. The client streams 16 kHz mono PCM16 chunks (base64) over a WebSocket; the server runs the transcription and streams live partial transcripts back.

Local TTS (Kokoro via sherpa-onnx OfflineTts) runs in the same worker process and is exposed as POST /api/dictation/tts/speak (JSON {text, speakerId?, speed?, model?} → WAV bytes; 503 with reasonCode while the model is downloading). TTS models live in the same catalog/downloader as STT models (local/model-catalog.js LOCAL_TTS_MODEL_CATALOG) and are managed by the same status/download/delete routes.

Ownership

runtime.js — registers GET /api/dictation/status, POST /api/dictation/models/:modelId/download, and the /api/dictation/ws WebSocket endpoint (auth-gated the same way as the terminal WS: UI session token or oc_url_token, plus origin check). Created from the startup pipeline (startup-pipeline-runtime.js) before the generic OpenCode proxy so routes are not shadowed.
stream-manager.js — DictationStreamManager, one per WS connection. Chunk reordering by seq + ack, resampling to the provider rate, auto-commit every ~15 s of audio, silence suppression by PCM peak, partial-transcript concatenation, adaptive finalization timeout.
service.js — provider resolution and readiness. Providers:
- local (default): sherpa-onnx Parakeet TDT in a forked worker process. Models auto-download in the background on first use; while missing, the stream fails with reasonCode: 'model_download_in_progress' and the status route reports per-model install/download state.
- openai-compatible: buffered per-segment transcription against any OpenAI-compatible /v1/audio/transcriptions endpoint (openai-compatible-session.js, reuses ../tts/stt.js).
local/ — worker process + client (IPC, idle shutdown TTL), sherpa recognizer engine and realtime session (throttled re-decode for partials), model catalog and downloader. The native sherpa-onnx-node addon is only ever loaded inside the worker process.
audio.js — PCM16 helpers: format parsing, peak, WAV wrapping, streaming linear resampler.

WebSocket protocol (JSON text frames)

Client → server: start {dictationId, format, options}, chunk {dictationId, seq, audio}, finish {dictationId, finalSeq}, cancel {dictationId}, ping.

Server → client: ready, ack {ackSeq}, partial {text}, finish_accepted {timeoutMs}, final {text}, error {error, retryable, reasonCode?}, pong.

options in start carries the client-selected provider config: { provider: 'local' | 'openai-compatible', language?, localModel?, openaiCompatible?: { baseUrl, model, apiKey } }.

Invariants

Never load sherpa-onnx-node in the main server process.
The stream manager acks only the highest contiguous seq; the client is expected to retain unacked segments for retry/replay.
Silence-only segments (peak < 300) are cleared, never committed, so Whisper-style providers do not hallucinate on silence.
Model files live under ~/.config/openchamber/speech-models.

3.1 KiB Raw Blame History

Dictation module

Ownership

WebSocket protocol (JSON text frames)

Invariants

3.1 KiB

Raw Blame History