BuzzbeeSCD/openchamber
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)

Browse Source 
Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).

Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
  seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
  segments auto-commit every ~15s with silence suppression and adaptive
  finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
  worker process — auto-download with progress, staged extraction with
  verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
  download/delete): Parakeet TDT v2 (English) and v3 (25 European
  languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
  cancel / insert / insert-and-send actions; failed transcriptions keep
  their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
  Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
  matching paddings/typography/gaps) — no layout shift when toggling

Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
  worker via /api/dictation/tts/speak, managed by the shared model
  pipeline; sentence-pipelined playback keeps time-to-first-audio at
  ~1 sentence regardless of message length, and stop cancels in-flight
  synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
  interword slashes aloud, and removes only absolute file paths

Settings:
- Voice page unified: a single read-aloud toggle owns all playback
  options (the confusing "Enable Voice Mode" is gone); a new "Enable
  voice input" toggle (default on, persisted to settings.json) hides
  the composer mic entirely when disabled

Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
  impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
  (https://localhost) was missing from the packaged-client allowlist,
  403-ing every WS connection — root cause of the old mobile SSE lock,
  which is now removed for all transports

Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
  explicitly validates the UI session and origin, with oc_url_token
  narrowly allowlisted and covered by tests; the dictation socket mints
  a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
  through runtimeFetch/getRuntimeUrlResolver, and runtime switches
  reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
  process in that runtime)

CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.

New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
Bohdan Triapitsyn
2026-07-04 02:48:07 +03:00
committed by GitHub
parent 3f5151d424
commit de1b85ac56
89 changed files with 8740 additions and 6061 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ui/src/components/chat/ChatInput.tsx
+53 -4
View File
@@ -1,6 +1,6 @@
import React from 'react';
import { Textarea } from '@/components/ui/textarea';
import { BrowserVoiceButton } from '@/components/voice';
import { ComposerDictation } from '@/components/dictation/ComposerDictation';
// sessionStore removed — currentSessionId comes from useSessionUIStore
import { useConfigStore } from '@/stores/useConfigStore';
import { useUIStore } from '@/stores/useUIStore';
@@ -381,7 +381,7 @@ const getProjectIconColor = (projectColor?: string | null): string | undefined =
};
const MemoModelControls = React.memo(ModelControls);
const MemoBrowserVoiceButton = React.memo(BrowserVoiceButton);
const MemoComposerDictation = React.memo(ComposerDictation);
const MemoMobileAgentButton = React.memo(MobileAgentButton);
const MemoMobileModelButton = React.memo(MobileModelButton);
const MemoStatusRow = React.memo(StatusRow);
@@ -2318,6 +2318,33 @@ const ChatInputComponent: React.FC<ChatInputProps> = ({ onOpenSettings, scrollTo
void handleSubmitRef.current();
}, []);
// Dictation: insert the transcript inline; optionally submit immediately.
// getCurrentInputSnapshot reads textareaRef.current.value first, so setting
// it synchronously lets handleSubmit pick up the text in the same tick.
const handleDictationInsert = React.useCallback((text: string) => {
setMessage((prev) => {
const next = appendInlineText(prev, text);
const textarea = textareaRef.current;
if (textarea) {
textarea.value = next;
}
return next;
});
setTimeout(() => {
textareaRef.current?.focus();
}, 0);
}, []);
const handleDictationInsertAndSend = React.useCallback((text: string) => {
const textarea = textareaRef.current;
const next = appendInlineText(textarea?.value ?? messageRef.current, text);
if (textarea) {
textarea.value = next;
}
setMessage(next);
void handleSubmitRef.current();
}, []);
// Preset chips rendered outside this component (e.g. under the welcome
// message on narrow surfaces) request a submit via the input store; consume
// it here so it routes through the same command-aware submit path.
@@ -4420,6 +4447,9 @@ const ChatInputComponent: React.FC<ChatInputProps> = ({ onOpenSettings, scrollTo
: undefined}
/>
)}
{/* Positioning context for the dictation overlay: covers the
text area + footer exactly, excluding MobileSessionStatusBar. */}
<div className={cn('relative flex flex-col', isDesktopExpanded && 'flex-1 min-h-0')}>
<div className={cn("overflow-hidden", isDesktopExpanded && 'flex flex-1 min-h-0 flex-col')}>
<div className="flex items-center gap-1 px-3 pt-1 flex-wrap relative z-10">
<AttachedVSCodeFileChips onShowPopup={handleShowAttachmentPreview} />
@@ -4559,7 +4589,16 @@ const ChatInputComponent: React.FC<ChatInputProps> = ({ onOpenSettings, scrollTo
/>
</div>
<div className="flex items-center gap-x-1 flex-shrink-0">
<MemoBrowserVoiceButton />
<MemoComposerDictation
radius={chatInputRadius}
isMobile={isMobile}
footerIconButtonClass={footerIconButtonClass}
footerPaddingClass={footerPaddingClass}
iconSizeClass={iconSizeClass}
sendIconSizeClass={sendIconSizeClass}
onInsert={handleDictationInsert}
onInsertAndSend={handleDictationInsertAndSend}
/>
<ComposerActionButtons
isMobile={isMobile}
footerIconButtonClass={footerIconButtonClass}
@@ -4614,7 +4653,16 @@ const ChatInputComponent: React.FC<ChatInputProps> = ({ onOpenSettings, scrollTo
</div>
<div className={cn('flex items-center flex-1 justify-end', footerGapClass, 'md:gap-x-3')}>
<MemoModelControls className={cn('flex-1 min-w-0 justify-end')} />
<MemoBrowserVoiceButton />
<MemoComposerDictation
radius={chatInputRadius}
isMobile={isMobile}
footerIconButtonClass={footerIconButtonClass}
footerPaddingClass={footerPaddingClass}
iconSizeClass={iconSizeClass}
sendIconSizeClass={sendIconSizeClass}
onInsert={handleDictationInsert}
onInsertAndSend={handleDictationInsertAndSend}
/>
<ComposerActionButtons
isMobile={isMobile}
footerIconButtonClass={footerIconButtonClass}
@@ -4633,6 +4681,7 @@ const ChatInputComponent: React.FC<ChatInputProps> = ({ onOpenSettings, scrollTo
</>
)}
</div>
</div>
{/* Mobile session panel: slide-up overlay toggled by MobileSessionPanelTrigger. */}
{isMobile && <MobileSessionStatusBar />}