Bohdan Triapitsyn
391f938334
Text-to-speech picked one voice regardless of what language a reply was in. A dependency-free language detector (script, marker letters, function words) now decides the language of the whole message once; with the new "Match the voice to the language of the text" setting the local provider switches to a catalog model for that language (Kokoro zh/en and Piper models for 12 languages, downloaded on first use like the existing model) and macOS say switches to an installed voice whose locale matches. The local voice picker lists voices of every installed model, and the settings show which language models are on disk. The Ukrainian Piper medium build is a character-level model that sherpa-onnx turns into noise, so the espeak-based Lada build is used instead. Claude-Session: https://claude.ai/code/session_017TK5JAYDfT3Fotc23UEg98
101 lines
5.2 KiB
Markdown
101 lines
5.2 KiB
Markdown
# Dictation module
|
|
|
|
Server-authoritative speech-to-text for the chat composer, plus local
|
|
text-to-speech. The client streams 16 kHz mono PCM16 chunks (base64) over a
|
|
WebSocket while the user speaks; the server buffers them and transcribes each
|
|
segment exactly once, when the segment is committed.
|
|
|
|
Transcription is deliberately not incremental. Parakeet is an offline model
|
|
trained on whole utterances, so re-decoding the growing buffer to animate a
|
|
live transcript costs O(n^2) work for a result the final decode replaces. The
|
|
composer shows no text while recording and inserts the full transcript on
|
|
stop.
|
|
|
|
Local TTS (Kokoro and Piper/VITS via sherpa-onnx OfflineTts) runs in the same
|
|
worker process and is exposed as `POST /api/dictation/tts/speak` (JSON
|
|
`{text, speakerId?, speed?, model?, language?, languageSample?}` → WAV bytes; 503 with
|
|
`reasonCode` while the model is downloading). TTS models live in the same
|
|
catalog/downloader as STT models (`local/model-catalog.js`
|
|
`LOCAL_TTS_MODEL_CATALOG`) and are managed by the same status/download/delete
|
|
routes.
|
|
|
|
Each TTS catalog entry declares the `languages` it speaks. With
|
|
`language: 'auto'` the service detects the language of `languageSample` — the
|
|
whole message the chunk belongs to, sent by the client with every chunk — or
|
|
of `text` when no sample is given
|
|
(`../tts/language-detect.js`, script plus function-word scoring, no
|
|
dependencies) and keeps the caller's model when it speaks that language;
|
|
otherwise it switches to the catalog model for the language, downloading it on
|
|
first use like any other model, and starts from that model's default speaker
|
|
(`defaultSpeakerByLanguage`) instead of the caller's speaker id. A language no
|
|
catalog model covers keeps the caller's model, so text is always spoken. The
|
|
response carries `X-Speech-Model` and `X-Speech-Language`.
|
|
|
|
## Ownership
|
|
|
|
- `runtime.js` — registers `GET /api/dictation/status`,
|
|
`POST /api/dictation/models/:modelId/download`, and the
|
|
`/api/dictation/ws` WebSocket endpoint (auth-gated the same way as the
|
|
terminal WS: UI session token or `oc_url_token`, plus origin check).
|
|
Created from the startup pipeline (`startup-pipeline-runtime.js`) before
|
|
the generic OpenCode proxy so routes are not shadowed.
|
|
- `stream-manager.js` — `DictationStreamManager`, one per WS connection.
|
|
Chunk reordering by `seq` + ack, resampling to the provider rate, segment
|
|
splitting, silence suppression by PCM peak, partial-transcript
|
|
concatenation, adaptive finalization timeout.
|
|
- `service.js` — provider resolution and readiness. Providers:
|
|
- `local` (default): sherpa-onnx Parakeet TDT in a forked worker process.
|
|
Models auto-download in the background on first use; while missing, the
|
|
stream fails with `reasonCode: 'model_download_in_progress'` and the
|
|
status route reports per-model install/download state.
|
|
- `openai-compatible`: buffered per-segment transcription against any
|
|
OpenAI-compatible `/v1/audio/transcriptions` endpoint
|
|
(`openai-compatible-session.js`, reuses `../tts/stt.js`).
|
|
- `local/` — worker process + client (IPC, idle shutdown TTL), sherpa
|
|
recognizer engine and segment session (one decode per committed segment),
|
|
model catalog and downloader. The native `sherpa-onnx-node` addon is only
|
|
ever loaded inside the worker process.
|
|
- `audio.js` — PCM16 helpers: format parsing, peak, WAV wrapping, streaming
|
|
linear resampler.
|
|
|
|
## WebSocket protocol (JSON text frames)
|
|
|
|
Client → server: `start {dictationId, format, options}`,
|
|
`chunk {dictationId, seq, audio}`, `finish {dictationId, finalSeq}`,
|
|
`cancel {dictationId}`, `ping`.
|
|
|
|
Server → client: `ready`, `ack {ackSeq}`, `partial {text}`,
|
|
`finish_accepted {timeoutMs}`, `final {text}`,
|
|
`error {error, retryable, reasonCode?}`, `pong`.
|
|
|
|
`options` in `start` carries the client-selected provider config:
|
|
`{ provider: 'local' | 'openai-compatible', language?, localModel?,
|
|
openaiCompatible?: { baseUrl, model, apiKey } }`.
|
|
|
|
## Segmentation
|
|
|
|
A dictation is one segment unless it runs long. Past `segmentMinSeconds`
|
|
(60 s) the manager commits on the first silent chunk, so cuts land at a pause
|
|
rather than mid-word; `segmentMaxSeconds` (90 s) is a hard cap for speech with
|
|
no pause in it. Client chunks are ~1 s, so "silent chunk" is roughly a second
|
|
of silence.
|
|
|
|
The bounds exist because Parakeet is a full-attention conformer: decode cost
|
|
and peak memory grow quadratically with segment length. Measured on Parakeet
|
|
v3 int8 with 2 threads: 60 s took 2.1 s and +90 MB, 180 s took 9.3 s and
|
|
+490 MB, 300 s took 21.3 s and +1.5 GB. Committed segments decode while the
|
|
user is still speaking, so only the tail is left to transcribe on stop.
|
|
|
|
## Invariants
|
|
|
|
- Never load `sherpa-onnx-node` in the main server process.
|
|
- Transcription happens on commit only; sessions never emit non-final
|
|
transcripts. The `partial` messages a client receives are the concatenation
|
|
of already-committed segments, and exist so a dictation that fails partway
|
|
can be salvaged instead of losing minutes of speech.
|
|
- The stream manager acks only the highest contiguous seq; the client is
|
|
expected to retain unacked segments for retry/replay.
|
|
- Silence-only segments (peak < 300) are cleared, never committed, so
|
|
Whisper-style providers do not hallucinate on silence.
|
|
- Model files live under `~/.config/openchamber/speech-models`.
|