Bohdan Triapitsyn
23928d342c
Parakeet is an offline model trained on whole utterances, so re-decoding the growing buffer to animate a live transcript cost O(n^2) work for a result the final decode replaced. Sessions now decode once per committed segment, and the composer shows a scrolling waveform of the mic level instead of running text. Long dictations split at a pause once past 60s (hard cap 90s) instead of on a blind 15s timer, so cuts no longer land mid-word. Committed segments decode while the user is still speaking: a 185s dictation returns 4.1s after stop instead of 11.0s, with identical text (816 vs 817 words). Also fixes two ways the stream manager could silently drop transcribed audio. It now counts the commits it issued instead of trusting the session's echoed events, so a commit still in flight when the client finishes can no longer be left out of the final text. And segment byte/peak accounting is reset where the commit is issued rather than when the event arrives, which could mistake the tail of a dictation for silence and clear it.
88 lines
4.4 KiB
Markdown
88 lines
4.4 KiB
Markdown
# Dictation module
|
|
|
|
Server-authoritative speech-to-text for the chat composer, plus local
|
|
text-to-speech. The client streams 16 kHz mono PCM16 chunks (base64) over a
|
|
WebSocket while the user speaks; the server buffers them and transcribes each
|
|
segment exactly once, when the segment is committed.
|
|
|
|
Transcription is deliberately not incremental. Parakeet is an offline model
|
|
trained on whole utterances, so re-decoding the growing buffer to animate a
|
|
live transcript costs O(n^2) work for a result the final decode replaces. The
|
|
composer shows no text while recording and inserts the full transcript on
|
|
stop.
|
|
|
|
Local TTS (Kokoro via sherpa-onnx OfflineTts) runs in the same worker process
|
|
and is exposed as `POST /api/dictation/tts/speak` (JSON `{text, speakerId?,
|
|
speed?, model?}` → WAV bytes; 503 with `reasonCode` while the model is
|
|
downloading). TTS models live in the same catalog/downloader as STT models
|
|
(`local/model-catalog.js` `LOCAL_TTS_MODEL_CATALOG`) and are managed by the
|
|
same status/download/delete routes.
|
|
|
|
## Ownership
|
|
|
|
- `runtime.js` — registers `GET /api/dictation/status`,
|
|
`POST /api/dictation/models/:modelId/download`, and the
|
|
`/api/dictation/ws` WebSocket endpoint (auth-gated the same way as the
|
|
terminal WS: UI session token or `oc_url_token`, plus origin check).
|
|
Created from the startup pipeline (`startup-pipeline-runtime.js`) before
|
|
the generic OpenCode proxy so routes are not shadowed.
|
|
- `stream-manager.js` — `DictationStreamManager`, one per WS connection.
|
|
Chunk reordering by `seq` + ack, resampling to the provider rate, segment
|
|
splitting, silence suppression by PCM peak, partial-transcript
|
|
concatenation, adaptive finalization timeout.
|
|
- `service.js` — provider resolution and readiness. Providers:
|
|
- `local` (default): sherpa-onnx Parakeet TDT in a forked worker process.
|
|
Models auto-download in the background on first use; while missing, the
|
|
stream fails with `reasonCode: 'model_download_in_progress'` and the
|
|
status route reports per-model install/download state.
|
|
- `openai-compatible`: buffered per-segment transcription against any
|
|
OpenAI-compatible `/v1/audio/transcriptions` endpoint
|
|
(`openai-compatible-session.js`, reuses `../tts/stt.js`).
|
|
- `local/` — worker process + client (IPC, idle shutdown TTL), sherpa
|
|
recognizer engine and segment session (one decode per committed segment),
|
|
model catalog and downloader. The native `sherpa-onnx-node` addon is only
|
|
ever loaded inside the worker process.
|
|
- `audio.js` — PCM16 helpers: format parsing, peak, WAV wrapping, streaming
|
|
linear resampler.
|
|
|
|
## WebSocket protocol (JSON text frames)
|
|
|
|
Client → server: `start {dictationId, format, options}`,
|
|
`chunk {dictationId, seq, audio}`, `finish {dictationId, finalSeq}`,
|
|
`cancel {dictationId}`, `ping`.
|
|
|
|
Server → client: `ready`, `ack {ackSeq}`, `partial {text}`,
|
|
`finish_accepted {timeoutMs}`, `final {text}`,
|
|
`error {error, retryable, reasonCode?}`, `pong`.
|
|
|
|
`options` in `start` carries the client-selected provider config:
|
|
`{ provider: 'local' | 'openai-compatible', language?, localModel?,
|
|
openaiCompatible?: { baseUrl, model, apiKey } }`.
|
|
|
|
## Segmentation
|
|
|
|
A dictation is one segment unless it runs long. Past `segmentMinSeconds`
|
|
(60 s) the manager commits on the first silent chunk, so cuts land at a pause
|
|
rather than mid-word; `segmentMaxSeconds` (90 s) is a hard cap for speech with
|
|
no pause in it. Client chunks are ~1 s, so "silent chunk" is roughly a second
|
|
of silence.
|
|
|
|
The bounds exist because Parakeet is a full-attention conformer: decode cost
|
|
and peak memory grow quadratically with segment length. Measured on Parakeet
|
|
v3 int8 with 2 threads: 60 s took 2.1 s and +90 MB, 180 s took 9.3 s and
|
|
+490 MB, 300 s took 21.3 s and +1.5 GB. Committed segments decode while the
|
|
user is still speaking, so only the tail is left to transcribe on stop.
|
|
|
|
## Invariants
|
|
|
|
- Never load `sherpa-onnx-node` in the main server process.
|
|
- Transcription happens on commit only; sessions never emit non-final
|
|
transcripts. The `partial` messages a client receives are the concatenation
|
|
of already-committed segments, and exist so a dictation that fails partway
|
|
can be salvaged instead of losing minutes of speech.
|
|
- The stream manager acks only the highest contiguous seq; the client is
|
|
expected to retain unacked segments for retry/replay.
|
|
- Silence-only segments (peak < 300) are cleared, never committed, so
|
|
Whisper-style providers do not hallucinate on silence.
|
|
- Model files live under `~/.config/openchamber/speech-models`.
|