feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)

Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).

Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
  seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
  segments auto-commit every ~15s with silence suppression and adaptive
  finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
  worker process — auto-download with progress, staged extraction with
  verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
  download/delete): Parakeet TDT v2 (English) and v3 (25 European
  languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
  cancel / insert / insert-and-send actions; failed transcriptions keep
  their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
  Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
  matching paddings/typography/gaps) — no layout shift when toggling

Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
  worker via /api/dictation/tts/speak, managed by the shared model
  pipeline; sentence-pipelined playback keeps time-to-first-audio at
  ~1 sentence regardless of message length, and stop cancels in-flight
  synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
  interword slashes aloud, and removes only absolute file paths

Settings:
- Voice page unified: a single read-aloud toggle owns all playback
  options (the confusing "Enable Voice Mode" is gone); a new "Enable
  voice input" toggle (default on, persisted to settings.json) hides
  the composer mic entirely when disabled

Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
  impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
  (https://localhost) was missing from the packaged-client allowlist,
  403-ing every WS connection — root cause of the old mobile SSE lock,
  which is now removed for all transports

Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
  explicitly validates the UI session and origin, with oc_url_token
  narrowly allowlisted and covered by tests; the dictation socket mints
  a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
  through runtimeFetch/getRuntimeUrlResolver, and runtime switches
  reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
  process in that runtime)

CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.

New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
Bohdan Triapitsyn
2026-07-04 02:48:07 +03:00
committed by GitHub
parent 3f5151d424
commit de1b85ac56
89 changed files with 8740 additions and 6061 deletions
@@ -721,10 +721,18 @@ export const createSettingsHelpers = (dependencies) => {
}
}
if (typeof candidate.dictationEnabled === 'boolean') {
result.dictationEnabled = candidate.dictationEnabled;
}
if (typeof candidate.sttProvider === 'string') {
const provider = candidate.sttProvider.trim();
if (provider === 'browser' || provider === 'server' || provider === 'wasm') {
if (provider === 'local' || provider === 'openai-compatible') {
result.sttProvider = provider;
} else if (provider === 'server') {
// Legacy provider migration: 'server' was the OpenAI-compatible endpoint.
result.sttProvider = 'openai-compatible';
} else if (provider === 'browser' || provider === 'wasm') {
result.sttProvider = 'local';
}
}
if (typeof candidate.sttServerUrl === 'string') {
@@ -739,10 +747,10 @@ export const createSettingsHelpers = (dependencies) => {
result.sttModel = trimmed;
}
}
if (typeof candidate.wasmSttModel === 'string') {
const trimmed = candidate.wasmSttModel.trim();
if (trimmed.length <= 256) {
result.wasmSttModel = trimmed;
if (typeof candidate.sttLocalModel === 'string') {
const trimmed = candidate.sttLocalModel.trim();
if (trimmed.length <= STT_MODEL_MAX_LENGTH) {
result.sttLocalModel = trimmed;
}
}
if (typeof candidate.sttLanguage === 'string') {
@@ -751,15 +759,6 @@ export const createSettingsHelpers = (dependencies) => {
result.sttLanguage = trimmed;
}
}
if (typeof candidate.sttSilenceThresholdDb === 'number' && Number.isFinite(candidate.sttSilenceThresholdDb)) {
result.sttSilenceThresholdDb = Math.max(-100, Math.min(0, candidate.sttSilenceThresholdDb));
}
if (typeof candidate.sttSilenceHoldMs === 'number' && Number.isFinite(candidate.sttSilenceHoldMs)) {
result.sttSilenceHoldMs = Math.max(250, Math.min(10000, Math.round(candidate.sttSilenceHoldMs)));
}
if (typeof candidate.sttTranscribeOnStop === 'boolean') {
result.sttTranscribeOnStop = candidate.sttTranscribeOnStop;
}
return result;
};
@@ -1,6 +1,7 @@
export const createStartupPipelineRuntime = (dependencies) => {
const {
createTerminalRuntime,
createDictationRuntime,
createMessageStreamWsRuntime,
createServerStartupRuntime,
} = dependencies;
@@ -52,6 +53,7 @@ export const createStartupPipelineRuntime = (dependencies) => {
tunnelRuntimeContext,
attachSignals,
apiOnly,
dictationModelsDir,
} = options;
const terminalRuntime = createTerminalRuntime({
@@ -71,6 +73,16 @@ export const createStartupPipelineRuntime = (dependencies) => {
TERMINAL_INPUT_WS_MAX_REBINDS_PER_WINDOW: terminalMaxRebindsPerWindow,
});
const dictationRuntime = createDictationRuntime({
app,
server,
express,
uiAuthController,
isRequestOriginAllowed,
rejectWebSocketUpgrade,
modelsDir: dictationModelsDir,
});
const messageStreamRuntime = createMessageStreamWsRuntime({
server,
uiAuthController,
@@ -125,6 +137,7 @@ export const createStartupPipelineRuntime = (dependencies) => {
return {
terminalRuntime,
dictationRuntime,
messageStreamRuntime,
};
};