feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)
Complete rebuild of voice input on a server-authoritative streaming architecture, replacing the legacy Web Speech / whole-blob / WASM engines and the dead voice-agent layer (~4k lines removed). Speech-to-text (dictation): - Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with seq/ack ordering; buffered audio is retained and replayed on reconnect - Server transcribes and streams live partial transcripts back; segments auto-commit every ~15s with silence suppression and adaptive finalization timeouts - Local provider (default, zero config): sherpa-onnx models in a forked worker process — auto-download with progress, staged extraction with verification, corrupt-model auto-recovery, idle shutdown after 5 min - Model catalog with settings picker (accuracy/speed ratings, sizes, download/delete): Parakeet TDT v2 (English) and v3 (25 European languages, auto-detected), Whisper base and tiny (multilingual, light) - OpenAI-compatible provider for any Whisper endpoint - Composer overlay with live transcript, volume meter, timer, and cancel / insert / insert-and-send actions; failed transcriptions keep their audio for retry or accepting the partial text as-is - Configurable keyboard shortcut (default mod+alt+v) toggles dictation; Enter confirms and Escape cancels while recording - Overlay is pixel-aligned with the composer (measured footer height, matching paddings/typography/gaps) — no layout shift when toggling Text-to-speech: - Local Kokoro provider (English, 11 voices) synthesized in the same worker via /api/dictation/tts/speak, managed by the shared model pipeline; sentence-pipelined playback keeps time-to-first-audio at ~1 sentence regardless of message length, and stop cancels in-flight synthesis - Sanitizer keeps inline-code content (strips backticks only), reads interword slashes aloud, and removes only absolute file paths Settings: - Voice page unified: a single read-aloud toggle owns all playback options (the confusing "Enable Voice Mode" is gone); a new "Enable voice input" toggle (default on, persisted to settings.json) hides the composer mic entirely when disabled Mobile and transport: - iOS/Android microphone permissions added (dictation was previously impossible on mobile) - Fixed Android WebSocket upgrades: the Capacitor WebView origin (https://localhost) was missing from the packaged-client allowlist, 403-ing every WS connection — root cause of the old mobile SSE lock, which is now removed for all transports Security and conventions: - All HTTP routes sit behind the global /api auth gate; the WS upgrade explicitly validates the UI session and origin, with oc_url_token narrowly allowlisted and covered by tests; the dictation socket mints a fresh URL token before connecting - Routes register before the generic OpenCode proxy; the client goes through runtimeFetch/getRuntimeUrlResolver, and runtime switches reset the dictation socket - VS Code deliberately reports dictation as unavailable (no server process in that runtime) CI: workflow Node bumped 20 -> 22 to match the repo engines and fix better-sqlite3 installs broken by node-gyp@latest on Node 20. New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64, Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
Bohdan Triapitsyn
committed by
GitHub
GitHub
parent
3f5151d424
commit
de1b85ac56
@@ -1743,8 +1743,6 @@ export const settingsDict = {
|
||||
'settings.voice.page.field.apiKeyHintUsingConfig': 'Użyto klucza z konfiguracji',
|
||||
'settings.voice.page.field.auto': 'Auto',
|
||||
'settings.voice.page.field.configuredAbove': 'Skonfigurowano powyżej',
|
||||
'settings.voice.page.field.enableVoiceMode': 'Włącz tryb głosowy',
|
||||
'settings.voice.page.field.enableVoiceModeAria': 'Włącz tryb głosowy',
|
||||
'settings.voice.page.field.language': 'Język',
|
||||
'settings.voice.page.field.messageReadAloudButton': 'Przycisk czytania wiadomości na głos',
|
||||
'settings.voice.page.field.messageReadAloudButtonAria': 'Przycisk czytania wiadomości na głos',
|
||||
@@ -1782,6 +1780,24 @@ export const settingsDict = {
|
||||
'settings.voice.page.provider.custom': 'Własny',
|
||||
'settings.voice.page.provider.say': 'Say',
|
||||
'settings.voice.page.provider.server': 'Serwer',
|
||||
'settings.voice.page.provider.local': 'Lokalny',
|
||||
'settings.voice.page.tooltip.sttLocal': 'Transkrypcja lokalna na serwerze OpenChamber. Modele pobierają się automatycznie; klucz API nie jest potrzebny.',
|
||||
'settings.voice.page.tooltip.localTts': 'Lokalna synteza na serwerze OpenChamber (Kokoro, angielski). Model pobiera się automatycznie; klucz API nie jest potrzebny.',
|
||||
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2 (angielski)',
|
||||
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3 (25 języków europejskich)',
|
||||
'settings.voice.page.stt.model.whisperBase': 'Whisper base (wielojęzyczny)',
|
||||
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny (wielojęzyczny)',
|
||||
'settings.voice.page.stt.badge.bestForEnglish': 'Najlepszy do angielskiego',
|
||||
'settings.voice.page.stt.badge.bestForMultilingual': 'Najlepszy wielojęzyczny',
|
||||
'settings.voice.page.stt.meta.accuracy': 'Dokładność',
|
||||
'settings.voice.page.stt.meta.speed': 'Szybkość',
|
||||
'settings.voice.page.stt.modelInstalled': 'Model zainstalowany',
|
||||
'settings.voice.page.stt.modelDownloading': 'Pobieranie modelu...',
|
||||
'settings.voice.page.stt.modelDownloadingProgress': 'Pobieranie modelu... {percent}%',
|
||||
'settings.voice.page.stt.modelNotInstalled': 'Model niepobrany',
|
||||
'settings.voice.page.stt.modelDownload': 'Pobierz',
|
||||
'settings.voice.page.stt.modelDelete': 'Usuń model',
|
||||
'settings.voice.page.stt.modelRetry': 'Ponów',
|
||||
'settings.voice.page.provider.wasm': 'Lokalny',
|
||||
'settings.voice.page.stt.wasmModel': 'Model Whisper',
|
||||
'settings.voice.page.stt.wasmLoaded': 'Model załadowany',
|
||||
@@ -1792,7 +1808,8 @@ export const settingsDict = {
|
||||
'settings.voice.page.stt.wasmRetry': 'Spróbuj ponownie',
|
||||
'settings.voice.page.section.playbackAndSummary': 'Odtwarzanie',
|
||||
'settings.voice.page.section.speechRecognition': 'Rozpoznawanie mowy',
|
||||
'settings.voice.page.section.voiceSetup': 'Konfiguracja głosu',
|
||||
'settings.voice.page.field.enableVoiceInput': 'Włącz wprowadzanie głosowe',
|
||||
'settings.voice.page.field.enableVoiceInputAria': 'Włącz wprowadzanie głosowe (dyktowanie)',
|
||||
'settings.voice.page.tooltip.browser': 'Darmowe, offline, ograniczone wsparcie mobilne.',
|
||||
'settings.voice.page.tooltip.custom': 'Serwer zgodny z OpenAI (na przykład Kokoro).',
|
||||
'settings.voice.page.tooltip.openai': 'Wysoka jakość, gotowe na urządzenia mobilne, wymaga klucza API.',
|
||||
|
||||
Reference in New Issue
Block a user