Files

T

Bohdan Triapitsyn 23928d342c feat(dictation): transcribe after recording instead of live

Parakeet is an offline model trained on whole utterances, so re-decoding
the growing buffer to animate a live transcript cost O(n^2) work for a
result the final decode replaced. Sessions now decode once per committed
segment, and the composer shows a scrolling waveform of the mic level
instead of running text.

Long dictations split at a pause once past 60s (hard cap 90s) instead of
on a blind 15s timer, so cuts no longer land mid-word. Committed segments
decode while the user is still speaking: a 185s dictation returns 4.1s
after stop instead of 11.0s, with identical text (816 vs 817 words).

Also fixes two ways the stream manager could silently drop transcribed
audio. It now counts the commits it issued instead of trusting the
session's echoed events, so a commit still in flight when the client
finishes can no longer be left out of the final text. And segment
byte/peak accounting is reset where the commit is issued rather than when
the event arrives, which could mistake the tail of a dictation for
silence and clear it.

2026-08-22 01:10:19 +03:00

4.4 KiB

Raw Blame History

Dictation module

Server-authoritative speech-to-text for the chat composer, plus local text-to-speech. The client streams 16 kHz mono PCM16 chunks (base64) over a WebSocket while the user speaks; the server buffers them and transcribes each segment exactly once, when the segment is committed.

Transcription is deliberately not incremental. Parakeet is an offline model trained on whole utterances, so re-decoding the growing buffer to animate a live transcript costs O(n^2) work for a result the final decode replaces. The composer shows no text while recording and inserts the full transcript on stop.

Local TTS (Kokoro via sherpa-onnx OfflineTts) runs in the same worker process and is exposed as POST /api/dictation/tts/speak (JSON {text, speakerId?, speed?, model?} → WAV bytes; 503 with reasonCode while the model is downloading). TTS models live in the same catalog/downloader as STT models (local/model-catalog.js LOCAL_TTS_MODEL_CATALOG) and are managed by the same status/download/delete routes.

Ownership

runtime.js — registers GET /api/dictation/status, POST /api/dictation/models/:modelId/download, and the /api/dictation/ws WebSocket endpoint (auth-gated the same way as the terminal WS: UI session token or oc_url_token, plus origin check). Created from the startup pipeline (startup-pipeline-runtime.js) before the generic OpenCode proxy so routes are not shadowed.
stream-manager.js — DictationStreamManager, one per WS connection. Chunk reordering by seq + ack, resampling to the provider rate, segment splitting, silence suppression by PCM peak, partial-transcript concatenation, adaptive finalization timeout.
service.js — provider resolution and readiness. Providers:
- local (default): sherpa-onnx Parakeet TDT in a forked worker process. Models auto-download in the background on first use; while missing, the stream fails with reasonCode: 'model_download_in_progress' and the status route reports per-model install/download state.
- openai-compatible: buffered per-segment transcription against any OpenAI-compatible /v1/audio/transcriptions endpoint (openai-compatible-session.js, reuses ../tts/stt.js).
local/ — worker process + client (IPC, idle shutdown TTL), sherpa recognizer engine and segment session (one decode per committed segment), model catalog and downloader. The native sherpa-onnx-node addon is only ever loaded inside the worker process.
audio.js — PCM16 helpers: format parsing, peak, WAV wrapping, streaming linear resampler.

WebSocket protocol (JSON text frames)

Client → server: start {dictationId, format, options}, chunk {dictationId, seq, audio}, finish {dictationId, finalSeq}, cancel {dictationId}, ping.

Server → client: ready, ack {ackSeq}, partial {text}, finish_accepted {timeoutMs}, final {text}, error {error, retryable, reasonCode?}, pong.

options in start carries the client-selected provider config: { provider: 'local' | 'openai-compatible', language?, localModel?, openaiCompatible?: { baseUrl, model, apiKey } }.

Segmentation

A dictation is one segment unless it runs long. Past segmentMinSeconds (60 s) the manager commits on the first silent chunk, so cuts land at a pause rather than mid-word; segmentMaxSeconds (90 s) is a hard cap for speech with no pause in it. Client chunks are ~1 s, so "silent chunk" is roughly a second of silence.

The bounds exist because Parakeet is a full-attention conformer: decode cost and peak memory grow quadratically with segment length. Measured on Parakeet v3 int8 with 2 threads: 60 s took 2.1 s and +90 MB, 180 s took 9.3 s and +490 MB, 300 s took 21.3 s and +1.5 GB. Committed segments decode while the user is still speaking, so only the tail is left to transcribe on stop.

Invariants

Never load sherpa-onnx-node in the main server process.
Transcription happens on commit only; sessions never emit non-final transcripts. The partial messages a client receives are the concatenation of already-committed segments, and exist so a dictation that fails partway can be salvaged instead of losing minutes of speech.
The stream manager acks only the highest contiguous seq; the client is expected to retain unacked segments for retry/replay.
Silence-only segments (peak < 300) are cleared, never committed, so Whisper-style providers do not hallucinate on silence.
Model files live under ~/.config/openchamber/speech-models.

4.4 KiB Raw Blame History