feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)
Complete rebuild of voice input on a server-authoritative streaming architecture, replacing the legacy Web Speech / whole-blob / WASM engines and the dead voice-agent layer (~4k lines removed). Speech-to-text (dictation): - Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with seq/ack ordering; buffered audio is retained and replayed on reconnect - Server transcribes and streams live partial transcripts back; segments auto-commit every ~15s with silence suppression and adaptive finalization timeouts - Local provider (default, zero config): sherpa-onnx models in a forked worker process — auto-download with progress, staged extraction with verification, corrupt-model auto-recovery, idle shutdown after 5 min - Model catalog with settings picker (accuracy/speed ratings, sizes, download/delete): Parakeet TDT v2 (English) and v3 (25 European languages, auto-detected), Whisper base and tiny (multilingual, light) - OpenAI-compatible provider for any Whisper endpoint - Composer overlay with live transcript, volume meter, timer, and cancel / insert / insert-and-send actions; failed transcriptions keep their audio for retry or accepting the partial text as-is - Configurable keyboard shortcut (default mod+alt+v) toggles dictation; Enter confirms and Escape cancels while recording - Overlay is pixel-aligned with the composer (measured footer height, matching paddings/typography/gaps) — no layout shift when toggling Text-to-speech: - Local Kokoro provider (English, 11 voices) synthesized in the same worker via /api/dictation/tts/speak, managed by the shared model pipeline; sentence-pipelined playback keeps time-to-first-audio at ~1 sentence regardless of message length, and stop cancels in-flight synthesis - Sanitizer keeps inline-code content (strips backticks only), reads interword slashes aloud, and removes only absolute file paths Settings: - Voice page unified: a single read-aloud toggle owns all playback options (the confusing "Enable Voice Mode" is gone); a new "Enable voice input" toggle (default on, persisted to settings.json) hides the composer mic entirely when disabled Mobile and transport: - iOS/Android microphone permissions added (dictation was previously impossible on mobile) - Fixed Android WebSocket upgrades: the Capacitor WebView origin (https://localhost) was missing from the packaged-client allowlist, 403-ing every WS connection — root cause of the old mobile SSE lock, which is now removed for all transports Security and conventions: - All HTTP routes sit behind the global /api auth gate; the WS upgrade explicitly validates the UI session and origin, with oc_url_token narrowly allowlisted and covered by tests; the dictation socket mints a fresh URL token before connecting - Routes register before the generic OpenCode proxy; the client goes through runtimeFetch/getRuntimeUrlResolver, and runtime switches reset the dictation socket - VS Code deliberately reports dictation as unavailable (no server process in that runtime) CI: workflow Node bumped 20 -> 22 to match the repo engines and fix better-sqlite3 installs broken by node-gyp@latest on Node 20. New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64, Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
committed by
GitHub
parent
3f5151d424
commit
de1b85ac56
@@ -0,0 +1,63 @@
|
||||
# Dictation module
|
||||
|
||||
Server-authoritative streaming speech-to-text for the chat composer, plus
|
||||
local text-to-speech. The client streams 16 kHz mono PCM16 chunks (base64)
|
||||
over a WebSocket; the server runs the transcription and streams live partial
|
||||
transcripts back.
|
||||
|
||||
Local TTS (Kokoro via sherpa-onnx OfflineTts) runs in the same worker process
|
||||
and is exposed as `POST /api/dictation/tts/speak` (JSON `{text, speakerId?,
|
||||
speed?, model?}` → WAV bytes; 503 with `reasonCode` while the model is
|
||||
downloading). TTS models live in the same catalog/downloader as STT models
|
||||
(`local/model-catalog.js` `LOCAL_TTS_MODEL_CATALOG`) and are managed by the
|
||||
same status/download/delete routes.
|
||||
|
||||
## Ownership
|
||||
|
||||
- `runtime.js` — registers `GET /api/dictation/status`,
|
||||
`POST /api/dictation/models/:modelId/download`, and the
|
||||
`/api/dictation/ws` WebSocket endpoint (auth-gated the same way as the
|
||||
terminal WS: UI session token or `oc_url_token`, plus origin check).
|
||||
Created from the startup pipeline (`startup-pipeline-runtime.js`) before
|
||||
the generic OpenCode proxy so routes are not shadowed.
|
||||
- `stream-manager.js` — `DictationStreamManager`, one per WS connection.
|
||||
Chunk reordering by `seq` + ack, resampling to the provider rate,
|
||||
auto-commit every ~15 s of audio, silence suppression by PCM peak,
|
||||
partial-transcript concatenation, adaptive finalization timeout.
|
||||
- `service.js` — provider resolution and readiness. Providers:
|
||||
- `local` (default): sherpa-onnx Parakeet TDT in a forked worker process.
|
||||
Models auto-download in the background on first use; while missing, the
|
||||
stream fails with `reasonCode: 'model_download_in_progress'` and the
|
||||
status route reports per-model install/download state.
|
||||
- `openai-compatible`: buffered per-segment transcription against any
|
||||
OpenAI-compatible `/v1/audio/transcriptions` endpoint
|
||||
(`openai-compatible-session.js`, reuses `../tts/stt.js`).
|
||||
- `local/` — worker process + client (IPC, idle shutdown TTL), sherpa
|
||||
recognizer engine and realtime session (throttled re-decode for partials),
|
||||
model catalog and downloader. The native `sherpa-onnx-node` addon is only
|
||||
ever loaded inside the worker process.
|
||||
- `audio.js` — PCM16 helpers: format parsing, peak, WAV wrapping, streaming
|
||||
linear resampler.
|
||||
|
||||
## WebSocket protocol (JSON text frames)
|
||||
|
||||
Client → server: `start {dictationId, format, options}`,
|
||||
`chunk {dictationId, seq, audio}`, `finish {dictationId, finalSeq}`,
|
||||
`cancel {dictationId}`, `ping`.
|
||||
|
||||
Server → client: `ready`, `ack {ackSeq}`, `partial {text}`,
|
||||
`finish_accepted {timeoutMs}`, `final {text}`,
|
||||
`error {error, retryable, reasonCode?}`, `pong`.
|
||||
|
||||
`options` in `start` carries the client-selected provider config:
|
||||
`{ provider: 'local' | 'openai-compatible', language?, localModel?,
|
||||
openaiCompatible?: { baseUrl, model, apiKey } }`.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Never load `sherpa-onnx-node` in the main server process.
|
||||
- The stream manager acks only the highest contiguous seq; the client is
|
||||
expected to retain unacked segments for retry/replay.
|
||||
- Silence-only segments (peak < 300) are cleared, never committed, so
|
||||
Whisper-style providers do not hallucinate on silence.
|
||||
- Model files live under `~/.config/openchamber/speech-models`.
|
||||
Reference in New Issue
Block a user