BuzzbeeSCD/openchamber
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)

Browse Source 
Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).

Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
  seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
  segments auto-commit every ~15s with silence suppression and adaptive
  finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
  worker process — auto-download with progress, staged extraction with
  verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
  download/delete): Parakeet TDT v2 (English) and v3 (25 European
  languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
  cancel / insert / insert-and-send actions; failed transcriptions keep
  their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
  Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
  matching paddings/typography/gaps) — no layout shift when toggling

Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
  worker via /api/dictation/tts/speak, managed by the shared model
  pipeline; sentence-pipelined playback keeps time-to-first-audio at
  ~1 sentence regardless of message length, and stop cancels in-flight
  synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
  interword slashes aloud, and removes only absolute file paths

Settings:
- Voice page unified: a single read-aloud toggle owns all playback
  options (the confusing "Enable Voice Mode" is gone); a new "Enable
  voice input" toggle (default on, persisted to settings.json) hides
  the composer mic entirely when disabled

Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
  impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
  (https://localhost) was missing from the packaged-client allowlist,
  403-ing every WS connection — root cause of the old mobile SSE lock,
  which is now removed for all transports

Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
  explicitly validates the UI session and origin, with oc_url_token
  narrowly allowlisted and covered by tests; the dictation socket mints
  a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
  through runtimeFetch/getRuntimeUrlResolver, and runtime switches
  reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
  process in that runtime)

CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.

New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
Bohdan Triapitsyn
2026-07-04 02:48:07 +03:00
committed by GitHub
parent 3f5151d424
commit de1b85ac56
89 changed files with 8740 additions and 6061 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ui/src/stores/useConfigStore.ts
+81 -128
View File
@@ -23,10 +23,6 @@ import { getSyncConfig, subscribeToSyncConfigChanges } from "@/sync/sync-refs";
const MODELS_DEV_API_URL = "https://models.dev/api.json";
const MODELS_DEV_PROXY_URL = "/api/openchamber/models-metadata";
const STT_SILENCE_THRESHOLD_DB_MIN = -100;
const STT_SILENCE_THRESHOLD_DB_MAX = 0;
const STT_SILENCE_HOLD_MS_MIN = 250;
const STT_SILENCE_HOLD_MS_MAX = 10000;
const FALLBACK_PROVIDER_ID = "opencode";
const FALLBACK_MODEL_ID = "big-pickle";
@@ -38,18 +34,19 @@ const GIT_UTILITY_PROVIDER_ID = "zen";
const GIT_UTILITY_PREFERRED_MODEL_ID = "big-pickle";
const PROVIDER_CONFIG_REFRESH_CONCURRENCY = 4;
const normalizeSttSilenceThresholdDb = (value: unknown): number | undefined => {
if (typeof value !== 'number' || !Number.isFinite(value)) {
return undefined;
const normalizeSttProvider = (value: unknown): 'local' | 'openai-compatible' | undefined => {
if (value === 'local' || value === 'openai-compatible') {
return value;
}
return Math.max(STT_SILENCE_THRESHOLD_DB_MIN, Math.min(STT_SILENCE_THRESHOLD_DB_MAX, value));
};
const normalizeSttSilenceHoldMs = (value: unknown): number | undefined => {
if (typeof value !== 'number' || !Number.isFinite(value)) {
return undefined;
// Legacy providers: 'server' used an OpenAI-compatible endpoint;
// 'browser' and 'wasm' map to the local default.
if (value === 'server') {
return 'openai-compatible';
}
return Math.max(STT_SILENCE_HOLD_MS_MIN, Math.min(STT_SILENCE_HOLD_MS_MAX, Math.round(value)));
if (value === 'browser' || value === 'wasm') {
return 'local';
}
return undefined;
};
interface OpenChamberDefaults {
@@ -61,13 +58,11 @@ interface OpenChamberDefaults {
defaultFileViewerPreview?: boolean;
zenModel?: string;
messageStreamTransport?: 'auto' | 'ws' | 'sse';
sttProvider?: 'browser' | 'server' | 'wasm';
sttProvider?: 'local' | 'openai-compatible';
sttServerUrl?: string;
wasmSttModel?: string;
sttModel?: string;
sttLocalModel?: string;
sttLanguage?: string;
sttSilenceThresholdDb?: number;
sttSilenceHoldMs?: number;
}
const fetchOpenChamberDefaults = async (): Promise<OpenChamberDefaults> => {
@@ -101,12 +96,11 @@ const fetchOpenChamberDefaults = async (): Promise<OpenChamberDefaults> => {
data?.messageStreamTransport === 'ws' || data?.messageStreamTransport === 'sse' || data?.messageStreamTransport === 'auto'
? data.messageStreamTransport
: undefined;
const sttProvider = data?.sttProvider === 'browser' || data?.sttProvider === 'server' || data?.sttProvider === 'wasm' ? data.sttProvider : undefined;
const sttProvider = normalizeSttProvider(data?.sttProvider);
const sttServerUrl = typeof data?.sttServerUrl === 'string' ? data.sttServerUrl.trim() : undefined;
const sttModel = typeof data?.sttModel === 'string' ? data.sttModel.trim() : undefined;
const sttLocalModel = typeof data?.sttLocalModel === 'string' ? data.sttLocalModel.trim() : undefined;
const sttLanguage = typeof data?.sttLanguage === 'string' ? data.sttLanguage.trim() : undefined;
const sttSilenceThresholdDb = normalizeSttSilenceThresholdDb(data?.sttSilenceThresholdDb);
const sttSilenceHoldMs = normalizeSttSilenceHoldMs(data?.sttSilenceHoldMs);
return finish('runtime-settings', {
defaultModel: defaultModel.length > 0 ? defaultModel : undefined,
@@ -120,9 +114,8 @@ const fetchOpenChamberDefaults = async (): Promise<OpenChamberDefaults> => {
sttProvider,
sttServerUrl,
sttModel,
sttLocalModel,
sttLanguage,
sttSilenceThresholdDb,
sttSilenceHoldMs,
});
}
} catch {
@@ -149,12 +142,11 @@ const fetchOpenChamberDefaults = async (): Promise<OpenChamberDefaults> => {
data?.messageStreamTransport === 'ws' || data?.messageStreamTransport === 'sse' || data?.messageStreamTransport === 'auto'
? data.messageStreamTransport
: undefined;
const sttProvider = data?.sttProvider === 'browser' || data?.sttProvider === 'server' ? data.sttProvider : undefined;
const sttProvider = normalizeSttProvider(data?.sttProvider);
const sttServerUrl = typeof data?.sttServerUrl === 'string' ? data.sttServerUrl.trim() : undefined;
const sttModel = typeof data?.sttModel === 'string' ? data.sttModel.trim() : undefined;
const sttLocalModel = typeof data?.sttLocalModel === 'string' ? data.sttLocalModel.trim() : undefined;
const sttLanguage = typeof data?.sttLanguage === 'string' ? data.sttLanguage.trim() : undefined;
const sttSilenceThresholdDb = normalizeSttSilenceThresholdDb(data?.sttSilenceThresholdDb);
const sttSilenceHoldMs = normalizeSttSilenceHoldMs(data?.sttSilenceHoldMs);
return finish('settings-route', {
defaultModel: defaultModel.length > 0 ? defaultModel : undefined,
@@ -168,9 +160,8 @@ const fetchOpenChamberDefaults = async (): Promise<OpenChamberDefaults> => {
sttProvider,
sttServerUrl,
sttModel,
sttLocalModel,
sttLanguage,
sttSilenceThresholdDb,
sttSilenceHoldMs,
});
} catch (error) {
markStartupTrace('config.defaults:error', { error: error instanceof Error ? error.message : String(error) });
@@ -989,33 +980,31 @@ interface ConfigStore {
settingsZenModel: string | undefined;
settingsMessageStreamTransport: 'auto' | 'ws' | 'sse';
// Voice provider preference ('browser', 'openai', 'openai-compatible', or 'say' for macOS)
voiceProvider: 'browser' | 'openai' | 'openai-compatible' | 'say';
setVoiceProvider: (provider: 'browser' | 'openai' | 'openai-compatible' | 'say') => void;
voiceProvider: 'browser' | 'local' | 'openai' | 'openai-compatible' | 'say';
setVoiceProvider: (provider: 'browser' | 'local' | 'openai' | 'openai-compatible' | 'say') => void;
// TTS settings
speechRate: number;
speechPitch: number;
speechVolume: number;
sayVoice: string;
browserVoice: string;
localTtsVoiceId: number;
openaiVoice: string;
openaiApiKey: string;
openaiCompatibleUrl: string;
openaiCompatibleApiKey: string;
openaiCompatibleVoice: string;
openaiCompatibleTtsModel: string;
// STT (speech-to-text) settings
sttProvider: 'browser' | 'server' | 'wasm';
// STT (dictation) settings
dictationEnabled: boolean;
sttProvider: 'local' | 'openai-compatible';
sttServerUrl: string;
sttApiKey: string;
sttModel: string;
wasmSttModel: string;
sttLocalModel: string;
sttLanguage: string;
sttSilenceThresholdDb: number;
sttSilenceHoldMs: number;
sttTranscribeOnStop: boolean;
showMessageTTSButtons: boolean;
ttsInputMode: 'sanitized' | 'raw';
voiceModeEnabled: boolean;
// Summarization settings
summarizeMessageTTS: boolean;
summarizeVoiceConversation: boolean;
@@ -1026,24 +1015,22 @@ interface ConfigStore {
setSpeechVolume: (volume: number) => void;
setSayVoice: (voice: string) => void;
setBrowserVoice: (voice: string) => void;
setLocalTtsVoiceId: (voiceId: number) => void;
setOpenaiVoice: (voice: string) => void;
setOpenaiApiKey: (apiKey: string) => void;
setOpenaiCompatibleUrl: (url: string) => void;
setOpenaiCompatibleApiKey: (apiKey: string) => void;
setOpenaiCompatibleVoice: (voice: string) => void;
setOpenaiCompatibleTtsModel: (model: string) => void;
setSttProvider: (provider: 'browser' | 'server' | 'wasm') => void;
setDictationEnabled: (enabled: boolean) => void;
setSttProvider: (provider: 'local' | 'openai-compatible') => void;
setSttServerUrl: (url: string) => void;
setSttApiKey: (apiKey: string) => void;
setSttModel: (model: string) => void;
setWasmSttModel: (model: string) => void;
setSttLocalModel: (model: string) => void;
setSttLanguage: (lang: string) => void;
setSttSilenceThresholdDb: (db: number) => void;
setSttSilenceHoldMs: (ms: number) => void;
setSttTranscribeOnStop: (enabled: boolean) => void;
setShowMessageTTSButtons: (show: boolean) => void;
setTtsInputMode: (mode: 'sanitized' | 'raw') => void;
setVoiceModeEnabled: (enabled: boolean) => void;
setSummarizeMessageTTS: (enabled: boolean) => void;
setSummarizeVoiceConversation: (enabled: boolean) => void;
setSummarizeCharacterThreshold: (threshold: number) => void;
@@ -1136,7 +1123,7 @@ export const useConfigStore = create<ConfigStore>()(
voiceProvider: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('voiceProvider');
if (saved === 'openai' || saved === 'browser' || saved === 'say' || saved === 'openai-compatible') return saved;
if (saved === 'openai' || saved === 'browser' || saved === 'local' || saved === 'say' || saved === 'openai-compatible') return saved;
}
return 'browser';
})(),
@@ -1179,6 +1166,17 @@ export const useConfigStore = create<ConfigStore>()(
}
return 'Samantha';
})(),
// Local (Kokoro) TTS speaker id - load from localStorage or default to 0
localTtsVoiceId: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('localTtsVoiceId');
if (saved !== null) {
const parsed = Number.parseInt(saved, 10);
if (Number.isInteger(parsed) && parsed >= 0) return parsed;
}
}
return 0;
})(),
// Browser voice - load from localStorage or default to empty (auto-select)
browserVoice: (() => {
if (typeof window !== 'undefined') {
@@ -1235,17 +1233,24 @@ export const useConfigStore = create<ConfigStore>()(
}
return 'kokoro';
})(),
// STT provider: 'browser' (Web Speech API), 'server' (OpenAI-compat), 'wasm' (local Whisper)
// Voice input (dictation) master toggle - default enabled
dictationEnabled: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('dictationEnabled');
if (saved === 'false') return false;
}
return true;
})(),
// STT provider: 'local' (server-side sherpa-onnx) or 'openai-compatible'
sttProvider: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('sttProvider');
if (saved === 'browser' || saved === 'server' || saved === 'wasm') return saved;
// Electron/Chromium's Web Speech API requires Google API keys
// not available in Electron, so default to WASM local Whisper.
const electron = (window as unknown as { __OPENCHAMBER_ELECTRON__?: { runtime?: string } }).__OPENCHAMBER_ELECTRON__;
if (electron?.runtime === 'electron') return 'wasm' as const;
if (saved === 'local' || saved === 'openai-compatible') return saved;
// Migrate legacy providers: 'server' used an OpenAI-compatible
// endpoint; 'browser' and 'wasm' map to the local default.
if (saved === 'server') return 'openai-compatible' as const;
}
return 'browser' as const;
return 'local' as const;
})(),
sttServerUrl: (() => {
if (typeof window !== 'undefined') {
@@ -1268,12 +1273,12 @@ export const useConfigStore = create<ConfigStore>()(
}
return 'deepdml/faster-whisper-large-v3-turbo-ct2';
})(),
wasmSttModel: (() => {
sttLocalModel: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('wasmSttModel');
const saved = localStorage.getItem('sttLocalModel');
if (saved) return saved;
}
return 'Xenova/whisper-base.en';
return 'parakeet-tdt-0.6b-v2-int8';
})(),
sttLanguage: (() => {
if (typeof window !== 'undefined') {
@@ -1282,33 +1287,6 @@ export const useConfigStore = create<ConfigStore>()(
}
return '';
})(),
sttSilenceThresholdDb: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('sttSilenceThresholdDb');
if (saved) {
const parsed = parseFloat(saved);
if (!isNaN(parsed)) return parsed;
}
}
return -45;
})(),
sttSilenceHoldMs: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('sttSilenceHoldMs');
if (saved) {
const parsed = parseInt(saved, 10);
if (!isNaN(parsed)) return parsed;
}
}
return 1500;
})(),
sttTranscribeOnStop: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('sttTranscribeOnStop');
if (saved === 'true') return true;
}
return false;
})(),
// Show TTS buttons on messages - disabled by default until user enables it
showMessageTTSButtons: (() => {
if (typeof window !== 'undefined') {
@@ -1324,14 +1302,6 @@ export const useConfigStore = create<ConfigStore>()(
}
return 'sanitized' as const;
})(),
// Voice mode enabled - load from localStorage or default to false
voiceModeEnabled: (() => {
if (typeof window !== 'undefined') {
const saved = localStorage.getItem('voiceModeEnabled');
if (saved === 'true') return true;
}
return false;
})(),
// Summarization settings
summarizeMessageTTS: (() => {
if (typeof window !== 'undefined') {
@@ -2060,9 +2030,8 @@ export const useConfigStore = create<ConfigStore>()(
sttProvider: openChamberDefaults.sttProvider ?? state.sttProvider,
sttServerUrl: openChamberDefaults.sttServerUrl ?? state.sttServerUrl,
sttModel: openChamberDefaults.sttModel ?? state.sttModel,
sttLocalModel: openChamberDefaults.sttLocalModel ?? state.sttLocalModel,
sttLanguage: openChamberDefaults.sttLanguage ?? state.sttLanguage,
sttSilenceThresholdDb: openChamberDefaults.sttSilenceThresholdDb ?? state.sttSilenceThresholdDb,
sttSilenceHoldMs: openChamberDefaults.sttSilenceHoldMs ?? state.sttSilenceHoldMs,
directoryScoped: {
...state.directoryScoped,
[directoryKey]: nextSnapshot,
@@ -2800,7 +2769,7 @@ export const useConfigStore = create<ConfigStore>()(
});
},
setVoiceProvider: (provider: 'browser' | 'openai' | 'openai-compatible' | 'say') => {
setVoiceProvider: (provider: 'browser' | 'local' | 'openai' | 'openai-compatible' | 'say') => {
set({ voiceProvider: provider });
if (typeof window !== 'undefined') {
localStorage.setItem('voiceProvider', provider);
@@ -2838,6 +2807,13 @@ export const useConfigStore = create<ConfigStore>()(
}
},
setLocalTtsVoiceId: (voiceId: number) => {
set({ localTtsVoiceId: voiceId });
if (typeof window !== 'undefined') {
localStorage.setItem('localTtsVoiceId', String(voiceId));
}
},
setBrowserVoice: (voice: string) => {
set({ browserVoice: voice });
if (typeof window !== 'undefined') {
@@ -2887,7 +2863,15 @@ export const useConfigStore = create<ConfigStore>()(
}
},
setSttProvider: (provider: 'browser' | 'server' | 'wasm') => {
setDictationEnabled: (enabled: boolean) => {
set({ dictationEnabled: enabled });
if (typeof window !== 'undefined') {
localStorage.setItem('dictationEnabled', String(enabled));
}
updateDesktopSettings({ dictationEnabled: enabled }).catch(() => {});
},
setSttProvider: (provider: 'local' | 'openai-compatible') => {
set({ sttProvider: provider });
if (typeof window !== 'undefined') {
localStorage.setItem('sttProvider', provider);
@@ -2918,12 +2902,12 @@ export const useConfigStore = create<ConfigStore>()(
updateDesktopSettings({ sttModel: model }).catch(() => {});
},
setWasmSttModel: (model: string) => {
set({ wasmSttModel: model });
setSttLocalModel: (model: string) => {
set({ sttLocalModel: model });
if (typeof window !== 'undefined') {
localStorage.setItem('wasmSttModel', model);
localStorage.setItem('sttLocalModel', model);
}
updateDesktopSettings({ wasmSttModel: model }).catch(() => {});
updateDesktopSettings({ sttLocalModel: model }).catch(() => {});
},
setSttLanguage: (lang: string) => {
@@ -2934,30 +2918,6 @@ export const useConfigStore = create<ConfigStore>()(
updateDesktopSettings({ sttLanguage: lang }).catch(() => {});
},
setSttSilenceThresholdDb: (db: number) => {
set({ sttSilenceThresholdDb: db });
if (typeof window !== 'undefined') {
localStorage.setItem('sttSilenceThresholdDb', String(db));
}
updateDesktopSettings({ sttSilenceThresholdDb: db }).catch(() => {});
},
setSttSilenceHoldMs: (ms: number) => {
set({ sttSilenceHoldMs: ms });
if (typeof window !== 'undefined') {
localStorage.setItem('sttSilenceHoldMs', String(ms));
}
updateDesktopSettings({ sttSilenceHoldMs: ms }).catch(() => {});
},
setSttTranscribeOnStop: (enabled: boolean) => {
set({ sttTranscribeOnStop: enabled });
if (typeof window !== 'undefined') {
localStorage.setItem('sttTranscribeOnStop', String(enabled));
}
updateDesktopSettings({ sttTranscribeOnStop: enabled }).catch(() => {});
},
setShowMessageTTSButtons: (show: boolean) => {
set({ showMessageTTSButtons: show });
if (typeof window !== 'undefined') {
@@ -2972,13 +2932,6 @@ export const useConfigStore = create<ConfigStore>()(
}
},
setVoiceModeEnabled: (enabled: boolean) => {
set({ voiceModeEnabled: enabled });
if (typeof window !== 'undefined') {
localStorage.setItem('voiceModeEnabled', String(enabled));
}
},
setSummarizeMessageTTS: (enabled: boolean) => {
set({ summarizeMessageTTS: enabled });
if (typeof window !== 'undefined') {