BuzzbeeSCD/openchamber
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(voice): first-class voice input and local TTS across web, desktop, and mobile (#2018)

Browse Source 
Complete rebuild of voice input on a server-authoritative streaming
architecture, replacing the legacy Web Speech / whole-blob / WASM engines
and the dead voice-agent layer (~4k lines removed).

Speech-to-text (dictation):
- Client streams 16 kHz mono PCM16 chunks over /api/dictation/ws with
  seq/ack ordering; buffered audio is retained and replayed on reconnect
- Server transcribes and streams live partial transcripts back;
  segments auto-commit every ~15s with silence suppression and adaptive
  finalization timeouts
- Local provider (default, zero config): sherpa-onnx models in a forked
  worker process — auto-download with progress, staged extraction with
  verification, corrupt-model auto-recovery, idle shutdown after 5 min
- Model catalog with settings picker (accuracy/speed ratings, sizes,
  download/delete): Parakeet TDT v2 (English) and v3 (25 European
  languages, auto-detected), Whisper base and tiny (multilingual, light)
- OpenAI-compatible provider for any Whisper endpoint
- Composer overlay with live transcript, volume meter, timer, and
  cancel / insert / insert-and-send actions; failed transcriptions keep
  their audio for retry or accepting the partial text as-is
- Configurable keyboard shortcut (default mod+alt+v) toggles dictation;
  Enter confirms and Escape cancels while recording
- Overlay is pixel-aligned with the composer (measured footer height,
  matching paddings/typography/gaps) — no layout shift when toggling

Text-to-speech:
- Local Kokoro provider (English, 11 voices) synthesized in the same
  worker via /api/dictation/tts/speak, managed by the shared model
  pipeline; sentence-pipelined playback keeps time-to-first-audio at
  ~1 sentence regardless of message length, and stop cancels in-flight
  synthesis
- Sanitizer keeps inline-code content (strips backticks only), reads
  interword slashes aloud, and removes only absolute file paths

Settings:
- Voice page unified: a single read-aloud toggle owns all playback
  options (the confusing "Enable Voice Mode" is gone); a new "Enable
  voice input" toggle (default on, persisted to settings.json) hides
  the composer mic entirely when disabled

Mobile and transport:
- iOS/Android microphone permissions added (dictation was previously
  impossible on mobile)
- Fixed Android WebSocket upgrades: the Capacitor WebView origin
  (https://localhost) was missing from the packaged-client allowlist,
  403-ing every WS connection — root cause of the old mobile SSE lock,
  which is now removed for all transports

Security and conventions:
- All HTTP routes sit behind the global /api auth gate; the WS upgrade
  explicitly validates the UI session and origin, with oc_url_token
  narrowly allowlisted and covered by tests; the dictation socket mints
  a fresh URL token before connecting
- Routes register before the generic OpenCode proxy; the client goes
  through runtimeFetch/getRuntimeUrlResolver, and runtime switches
  reset the dictation socket
- VS Code deliberately reports dictation as unavailable (no server
  process in that runtime)

CI: workflow Node bumped 20 -> 22 to match the repo engines and fix
better-sqlite3 installs broken by node-gyp@latest on Node 20.

New dependency: sherpa-onnx-node (prebuilt N-API; macOS/Linux x64+arm64,
Windows x64 — Windows-on-ARM falls back to the OpenAI-compatible provider)
This commit is contained in:
Bohdan Triapitsyn
2026-07-04 02:48:07 +03:00
committed by GitHub
parent 3f5151d424
commit de1b85ac56
89 changed files with 8740 additions and 6061 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ui/src/lib/desktop.ts
+3 -5
View File
@@ -178,14 +178,12 @@ export type DesktopSettings = {
responseStyleEnabled?: boolean;
responseStylePreset?: 'concise' | 'detailed' | 'mentor' | 'pushback' | 'noFiller' | 'matchEnergy' | 'warmPeer' | 'custom';
responseStyleCustomInstructions?: string;
sttProvider?: 'browser' | 'server' | 'wasm';
dictationEnabled?: boolean;
sttProvider?: 'local' | 'openai-compatible';
sttServerUrl?: string;
sttModel?: string;
wasmSttModel?: string;
sttLocalModel?: string;
sttLanguage?: string;
sttSilenceThresholdDb?: number;
sttSilenceHoldMs?: number;
sttTranscribeOnStop?: boolean;
// Global draft welcome starters (pinned commands/skills), persisted to settings.json
draftStarters?: DraftStarterRef[];
};
packages/ui/src/lib/dictation/dictation-client.ts
+447
View File
@@ -0,0 +1,447 @@
/**
* WebSocket client for the OpenChamber dictation endpoint (/api/dictation/ws).
*
* One shared client per app. The socket is opened lazily when a dictation
* starts and closed after an idle delay. URLs are resolved at connect time via
* the runtime URL resolver so runtime switches never leak a stale endpoint.
*/
import { getRuntimeUrlResolver } from '@/lib/runtime-url';
import { refreshRuntimeUrlAuthToken } from '@/lib/runtime-auth';
export interface DictationStartOptions {
provider?: 'local' | 'openai-compatible';
language?: string;
localModel?: string;
openaiCompatible?: {
baseUrl?: string;
model?: string;
apiKey?: string;
};
}
interface DictationServerMessage {
type: string;
dictationId?: string;
ackSeq?: number;
text?: string;
timeoutMs?: number;
error?: string;
retryable?: boolean;
reasonCode?: string;
}
interface DictationStreamError extends Error {
retryable: boolean;
reasonCode?: string;
}
const createStreamError = (message: string, retryable: boolean, reasonCode?: string): DictationStreamError => {
const error = new Error(message) as DictationStreamError;
error.name = 'DictationStreamError';
error.retryable = retryable;
if (reasonCode) {
error.reasonCode = reasonCode;
}
return error;
};
const CONNECT_TIMEOUT_MS = 10000;
const START_TIMEOUT_MS = 15000;
const IDLE_CLOSE_DELAY_MS = 30000;
const DEFAULT_FINISH_TIMEOUT_MS = 30000;
type ConnectionStatusListener = (connected: boolean) => void;
type PartialListener = (dictationId: string, text: string) => void;
interface PendingStart {
resolve: () => void;
reject: (error: Error) => void;
timeout: ReturnType<typeof setTimeout>;
}
interface PendingFinish {
resolve: (result: { text: string }) => void;
reject: (error: Error) => void;
timeout: ReturnType<typeof setTimeout> | null;
}
export class DictationClient {
private socket: WebSocket | null = null;
private connectPromise: Promise<void> | null = null;
private idleCloseTimer: ReturnType<typeof setTimeout> | null = null;
private readonly pendingStarts = new Map<string, PendingStart>();
private readonly pendingFinishes = new Map<string, PendingFinish>();
private readonly connectionListeners = new Set<ConnectionStatusListener>();
private readonly partialListeners = new Set<PartialListener>();
private activeDictations = 0;
get isConnected(): boolean {
return this.socket?.readyState === WebSocket.OPEN;
}
subscribeConnectionStatus(listener: ConnectionStatusListener): () => void {
this.connectionListeners.add(listener);
return () => {
this.connectionListeners.delete(listener);
};
}
onPartial(listener: PartialListener): () => void {
this.partialListeners.add(listener);
return () => {
this.partialListeners.delete(listener);
};
}
async ensureConnected(): Promise<void> {
if (this.isConnected) {
return;
}
if (this.connectPromise) {
await this.connectPromise;
return;
}
// A WebSocket upgrade can't carry an Authorization header, so it
// authenticates via the oc_url_token query param. Mint/await a valid
// token BEFORE connecting — the sync getter returns "" while the token
// is unminted or inside its expiry skew, and the server would reject
// the upgrade with 401.
try {
await refreshRuntimeUrlAuthToken();
} catch {
// No auth configured (local runtime) — proceed without a token.
}
this.connectPromise = new Promise<void>((resolve, reject) => {
let settled = false;
let socket: WebSocket;
try {
const url = getRuntimeUrlResolver().websocket('/api/dictation/ws');
socket = new WebSocket(url);
} catch (error) {
this.connectPromise = null;
reject(error instanceof Error ? error : new Error(String(error)));
return;
}
const timeout = setTimeout(() => {
if (!settled) {
settled = true;
this.connectPromise = null;
try {
socket.close();
} catch {
// ignore
}
reject(new Error('Dictation connection timed out'));
}
}, CONNECT_TIMEOUT_MS);
socket.onopen = () => {
// Wait for the server 'ready' frame before resolving.
};
socket.onmessage = (event) => {
let message: DictationServerMessage;
try {
message = JSON.parse(String(event.data));
} catch {
return;
}
if (!settled && message.type === 'ready') {
settled = true;
clearTimeout(timeout);
this.socket = socket;
this.connectPromise = null;
this.notifyConnection(true);
resolve();
return;
}
this.handleMessage(message);
};
socket.onerror = () => {
if (!settled) {
settled = true;
clearTimeout(timeout);
this.connectPromise = null;
reject(new Error('Dictation connection failed'));
}
};
socket.onclose = () => {
if (!settled) {
settled = true;
clearTimeout(timeout);
this.connectPromise = null;
reject(new Error('Dictation connection closed'));
return;
}
if (this.socket === socket) {
this.socket = null;
this.rejectAllPending(new Error('Dictation connection lost'));
this.notifyConnection(false);
}
};
});
await this.connectPromise;
}
/**
* Start a dictation stream. Resolves once the server acks the stream.
*/
async startDictationStream(
dictationId: string,
format: string,
options: DictationStartOptions,
): Promise<void> {
await this.ensureConnected();
this.activeDictations += 1;
this.clearIdleCloseTimer();
return new Promise<void>((resolve, reject) => {
const timeout = setTimeout(() => {
this.pendingStarts.delete(dictationId);
this.releaseDictation();
reject(new Error('Dictation start timed out'));
}, START_TIMEOUT_MS);
this.pendingStarts.set(dictationId, {
resolve: () => {
clearTimeout(timeout);
resolve();
},
reject: (error) => {
clearTimeout(timeout);
this.releaseDictation();
reject(error);
},
timeout });
if (!this.send({ type: 'start', dictationId, format, options })) {
clearTimeout(timeout);
this.pendingStarts.delete(dictationId);
this.releaseDictation();
reject(new Error('Dictation connection lost'));
}
});
}
sendDictationStreamChunk(dictationId: string, seq: number, audioBase64: string): boolean {
return this.send({ type: 'chunk', dictationId, seq, audio: audioBase64 });
}
/**
* Finish a dictation stream. Resolves with the final transcript.
*/
finishDictationStream(dictationId: string, finalSeq: number): Promise<{ text: string }> {
return new Promise<{ text: string }>((resolve, reject) => {
const pending: PendingFinish = {
resolve: (result) => {
if (pending.timeout) {
clearTimeout(pending.timeout);
}
this.releaseDictation();
resolve(result);
},
reject: (error) => {
if (pending.timeout) {
clearTimeout(pending.timeout);
}
this.releaseDictation();
reject(error);
},
timeout: null,
};
pending.timeout = setTimeout(() => {
this.pendingFinishes.delete(dictationId);
this.releaseDictation();
reject(new Error('Timed out waiting for transcription'));
}, DEFAULT_FINISH_TIMEOUT_MS);
this.pendingFinishes.set(dictationId, pending);
if (!this.send({ type: 'finish', dictationId, finalSeq })) {
this.pendingFinishes.delete(dictationId);
pending.reject(new Error('Dictation connection lost'));
}
});
}
cancelDictationStream(dictationId: string): void {
this.send({ type: 'cancel', dictationId });
const start = this.pendingStarts.get(dictationId);
if (start) {
this.pendingStarts.delete(dictationId);
start.reject(new Error('Dictation cancelled'));
}
const finish = this.pendingFinishes.get(dictationId);
if (finish) {
this.pendingFinishes.delete(dictationId);
finish.reject(new Error('Dictation cancelled'));
}
this.releaseDictation();
this.scheduleIdleCloseIfReady();
}
private handleMessage(message: DictationServerMessage): void {
const dictationId = message.dictationId;
if (!dictationId) {
return;
}
switch (message.type) {
case 'ack': {
const pendingStart = this.pendingStarts.get(dictationId);
if (pendingStart) {
this.pendingStarts.delete(dictationId);
pendingStart.resolve();
}
return;
}
case 'partial': {
for (const listener of this.partialListeners) {
listener(dictationId, message.text ?? '');
}
return;
}
case 'finish_accepted': {
const pendingFinish = this.pendingFinishes.get(dictationId);
if (pendingFinish && typeof message.timeoutMs === 'number') {
if (pendingFinish.timeout) {
clearTimeout(pendingFinish.timeout);
}
pendingFinish.timeout = setTimeout(() => {
this.pendingFinishes.delete(dictationId);
pendingFinish.reject(new Error('Timed out waiting for transcription'));
}, message.timeoutMs + 5000);
}
return;
}
case 'final': {
const pendingFinish = this.pendingFinishes.get(dictationId);
if (pendingFinish) {
this.pendingFinishes.delete(dictationId);
pendingFinish.resolve({ text: message.text ?? '' });
}
this.scheduleIdleCloseIfReady();
return;
}
case 'error': {
const error = createStreamError(
message.error || 'Dictation failed',
message.retryable !== false,
message.reasonCode,
);
const pendingStart = this.pendingStarts.get(dictationId);
if (pendingStart) {
this.pendingStarts.delete(dictationId);
pendingStart.reject(error);
}
const pendingFinish = this.pendingFinishes.get(dictationId);
if (pendingFinish) {
this.pendingFinishes.delete(dictationId);
pendingFinish.reject(error);
}
this.scheduleIdleCloseIfReady();
return;
}
default:
}
}
private send(message: object): boolean {
if (!this.isConnected || !this.socket) {
return false;
}
try {
this.socket.send(JSON.stringify(message));
return true;
} catch {
return false;
}
}
private notifyConnection(connected: boolean): void {
for (const listener of this.connectionListeners) {
listener(connected);
}
}
private rejectAllPending(error: Error): void {
for (const [dictationId, pending] of this.pendingStarts) {
this.pendingStarts.delete(dictationId);
pending.reject(error);
}
for (const [dictationId, pending] of this.pendingFinishes) {
this.pendingFinishes.delete(dictationId);
pending.reject(error);
}
this.activeDictations = 0;
}
private releaseDictation(): void {
this.activeDictations = Math.max(0, this.activeDictations - 1);
this.scheduleIdleCloseIfReady();
}
private scheduleIdleCloseIfReady(): void {
if (this.activeDictations > 0 || this.pendingFinishes.size > 0 || this.pendingStarts.size > 0) {
return;
}
this.clearIdleCloseTimer();
this.idleCloseTimer = setTimeout(() => {
if (this.activeDictations === 0 && this.pendingFinishes.size === 0 && this.pendingStarts.size === 0) {
const socket = this.socket;
this.socket = null;
if (socket) {
try {
socket.close(1000, 'idle');
} catch {
// ignore
}
this.notifyConnection(false);
}
}
}, IDLE_CLOSE_DELAY_MS);
}
/**
* Runtime switch: close the socket and fail all in-flight dictations so
* nothing keeps streaming to the previous runtime.
*/
cancelAllForRuntimeSwitch(): void {
this.clearIdleCloseTimer();
const socket = this.socket;
this.socket = null;
this.connectPromise = null;
this.rejectAllPending(new Error('Runtime changed'));
if (socket) {
try {
socket.close(1000, 'runtime switch');
} catch {
// ignore
}
this.notifyConnection(false);
}
}
private clearIdleCloseTimer(): void {
if (this.idleCloseTimer) {
clearTimeout(this.idleCloseTimer);
this.idleCloseTimer = null;
}
}
}
export const dictationClient = new DictationClient();
if (typeof window !== 'undefined') {
window.addEventListener('openchamber:runtime-endpoint-changed', () => {
// Drop the socket so the next dictation reconnects to the new runtime.
dictationClient.cancelAllForRuntimeSwitch();
});
}
packages/ui/src/lib/dictation/dictation-stream-sender.ts
+240
View File
@@ -0,0 +1,240 @@
/**
* Small, non-React state machine for dictation streaming.
*
* Responsibilities:
* - Maintain an ordered buffer of base64 PCM segments
* - Start/restart a dictation stream (dictationId)
* - Send missing segments (seq) when connected
* - Finish/cancel the stream
*
* Segments are retained until the dictation completes, which enables replay
* after a connection drop (`resetStreamForReplay()` + `finish()`).
*/
import type { DictationClient, DictationStartOptions } from './dictation-client';
const MAX_CHUNKS_PER_FLUSH_TURN = 128;
const PCM_DICTATION_FORMAT = 'audio/pcm;rate=16000;bits=16';
const waitForNextFlushTurn = (): Promise<void> => new Promise((resolve) => setTimeout(resolve, 0));
const createDictationIdDefault = (): string => {
const rand = Math.random().toString(36).slice(2, 10);
return `dic_${Date.now().toString(16)}${rand}`;
};
export interface DictationFinishResult {
dictationId: string;
text: string;
}
export class DictationStreamSender {
private readonly client: DictationClient;
private readonly format: string;
private readonly createDictationId: () => string;
private getStartOptions: () => DictationStartOptions;
private dictationId: string | null = null;
private sendSeq = 0;
private segments: string[] = [];
private streamReady = false;
private flushTimer: ReturnType<typeof setTimeout> | null = null;
private drainWaiters: Array<() => void> = [];
private startGeneration = 0;
private startPromise: Promise<void> | null = null;
constructor(params: {
client: DictationClient;
getStartOptions: () => DictationStartOptions;
format?: string;
createDictationId?: () => string;
}) {
this.client = params.client;
this.format = params.format ?? PCM_DICTATION_FORMAT;
this.getStartOptions = params.getStartOptions;
this.createDictationId = params.createDictationId ?? createDictationIdDefault;
}
getDictationId(): string | null {
return this.dictationId;
}
getFinalSeq(): number {
return this.segments.length - 1;
}
hasSegments(): boolean {
return this.segments.length > 0;
}
clearAll(): void {
this.clearScheduledFlush();
this.dictationId = null;
this.sendSeq = 0;
this.segments = [];
this.streamReady = false;
this.startPromise = null;
this.startGeneration += 1;
}
resetStreamForReplay(): void {
this.clearScheduledFlush();
this.dictationId = null;
this.sendSeq = 0;
this.streamReady = false;
this.startPromise = null;
this.startGeneration += 1;
}
enqueueSegment(base64Pcm: string): void {
this.segments.push(base64Pcm);
if (!this.client.isConnected) {
return;
}
if (!this.dictationId) {
if (!this.startPromise) {
void this.restartStream().catch(() => {
// Start failures surface through finish(); segments are retained.
});
}
return;
}
this.flush();
}
flush(): number {
const dictationId = this.dictationId;
if (!this.client.isConnected || !dictationId || !this.streamReady) {
return 0;
}
let sent = 0;
while (this.sendSeq < this.segments.length && sent < MAX_CHUNKS_PER_FLUSH_TURN) {
const seq = this.sendSeq;
const audio = this.segments[seq];
if (!this.client.sendDictationStreamChunk(dictationId, seq, audio)) {
break;
}
this.sendSeq = seq + 1;
sent += 1;
}
if (this.hasPendingSegments()) {
this.scheduleFlush();
} else {
this.resolveDrainWaiters();
}
return sent;
}
async restartStream(): Promise<void> {
this.startGeneration += 1;
const generation = this.startGeneration;
const dictationId = this.createDictationId();
this.dictationId = dictationId;
this.sendSeq = 0;
this.streamReady = false;
const start = (async () => {
await this.client.startDictationStream(dictationId, this.format, this.getStartOptions());
if (this.startGeneration !== generation) {
return;
}
if (this.dictationId !== dictationId) {
return;
}
this.streamReady = true;
this.flush();
})()
.catch((error) => {
// Keep segments for retry, but clear the stream so finish can error cleanly.
if (this.startGeneration === generation && this.dictationId === dictationId) {
this.dictationId = null;
this.streamReady = false;
}
throw error;
})
.finally(() => {
if (this.startPromise === start) {
this.startPromise = null;
}
});
this.startPromise = start;
await start;
}
async finish(finalSeq: number): Promise<DictationFinishResult> {
if (!this.dictationId) {
await this.restartStream();
}
if (this.startPromise) {
await this.startPromise;
}
const dictationId = this.dictationId;
if (!dictationId || !this.streamReady) {
throw new Error('Failed to start dictation stream');
}
this.flush();
await this.waitForFlushDrain();
const result = await this.client.finishDictationStream(dictationId, finalSeq);
return { dictationId, text: result.text };
}
cancel(): void {
const dictationId = this.dictationId;
if (this.client.isConnected && dictationId) {
this.client.cancelDictationStream(dictationId);
}
this.resetStreamForReplay();
}
private hasPendingSegments(): boolean {
return this.sendSeq < this.segments.length;
}
private scheduleFlush(): void {
if (this.flushTimer) {
return;
}
this.flushTimer = setTimeout(() => {
this.flushTimer = null;
this.flush();
}, 0);
}
private clearScheduledFlush(): void {
if (!this.flushTimer) {
return;
}
clearTimeout(this.flushTimer);
this.flushTimer = null;
}
private async waitForFlushDrain(): Promise<void> {
while (this.hasPendingSegments()) {
if (!this.client.isConnected || !this.dictationId || !this.streamReady) {
throw new Error('Failed to flush dictation stream');
}
await new Promise<void>((resolve) => {
this.drainWaiters.push(resolve);
});
await waitForNextFlushTurn();
}
}
private resolveDrainWaiters(): void {
const waiters = this.drainWaiters;
this.drainWaiters = [];
for (const resolve of waiters) {
resolve();
}
}
}
packages/ui/src/lib/dictation/use-dictation-audio-source.ts
+301
View File
@@ -0,0 +1,301 @@
/**
* Microphone capture for dictation.
*
* Captures mono audio via getUserMedia, taps it with a ScriptProcessorNode
* (universally supported, including iOS WKWebView), resamples Float32 to
* 16 kHz PCM16LE, and emits ~1-second base64 chunks plus a normalized RMS
* volume for the level meter.
*/
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
export interface DictationAudioSourceConfig {
onPcmSegment: (base64Pcm: string) => void;
onError?: (error: Error) => void;
}
export interface DictationAudioSource {
start: () => Promise<void>;
stop: () => Promise<void>;
volume: number;
}
const OUTPUT_RATE = 16000;
const CHUNK_SAMPLES = OUTPUT_RATE; // ~1s per chunk
const getAudioContextCtor = (): typeof AudioContext | null => {
if (typeof window === 'undefined') {
return null;
}
const win = window as typeof window & { webkitAudioContext?: typeof AudioContext };
return win.AudioContext || win.webkitAudioContext || null;
};
const floatToInt16 = (sample: number): number => {
const clamped = Math.max(-1, Math.min(1, sample));
return clamped < 0 ? Math.round(clamped * 0x8000) : Math.round(clamped * 0x7fff);
};
const resampleToPcm16 = (input: Float32Array, inputRate: number, outputRate: number): Int16Array => {
if (input.length === 0) {
return new Int16Array(0);
}
if (inputRate === outputRate) {
const out = new Int16Array(input.length);
for (let i = 0; i < input.length; i++) {
out[i] = floatToInt16(input[i]);
}
return out;
}
const ratio = inputRate / outputRate;
const outputLength = Math.max(1, Math.round(input.length / ratio));
const out = new Int16Array(outputLength);
for (let i = 0; i < outputLength; i++) {
const sourceIndex = i * ratio;
const i0 = Math.floor(sourceIndex);
const i1 = Math.min(input.length - 1, i0 + 1);
const frac = sourceIndex - i0;
out[i] = floatToInt16(input[i0] * (1 - frac) + input[i1] * frac);
}
return out;
};
const concatInt16 = (a: Int16Array, b: Int16Array): Int16Array => {
if (a.length === 0) {
return b;
}
if (b.length === 0) {
return a;
}
const out = new Int16Array(a.length + b.length);
out.set(a, 0);
out.set(b, a.length);
return out;
};
const int16ToBase64 = (pcm: Int16Array): string => {
const bytes = new Uint8Array(pcm.buffer, pcm.byteOffset, pcm.byteLength);
let binary = '';
for (let i = 0; i < bytes.length; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
};
interface CaptureGraph {
stream: MediaStream | null;
context: AudioContext | null;
source: MediaStreamAudioSourceNode | null;
processor: ScriptProcessorNode | null;
gain: GainNode | null;
pending: Int16Array;
started: boolean;
}
const emptyGraph = (): CaptureGraph => ({
stream: null,
context: null,
source: null,
processor: null,
gain: null,
pending: new Int16Array(0),
started: false,
});
const safeDisconnect = (node: AudioNode | null): void => {
if (!node) {
return;
}
try {
node.disconnect();
} catch {
// no-op
}
};
export const isDictationCaptureSupported = (): boolean => {
if (typeof navigator === 'undefined' || typeof window === 'undefined') {
return false;
}
if (!navigator.mediaDevices || typeof navigator.mediaDevices.getUserMedia !== 'function') {
return false;
}
return getAudioContextCtor() !== null;
};
export function useDictationAudioSource(config: DictationAudioSourceConfig): DictationAudioSource {
const [volume, setVolume] = useState(0);
const onPcmSegmentRef = useRef(config.onPcmSegment);
const onErrorRef = useRef(config.onError);
useEffect(() => {
onPcmSegmentRef.current = config.onPcmSegment;
onErrorRef.current = config.onError;
}, [config.onPcmSegment, config.onError]);
const graphRef = useRef<CaptureGraph>(emptyGraph());
const start = useCallback(async () => {
if (graphRef.current.started) {
return;
}
if (
typeof navigator === 'undefined' ||
!navigator.mediaDevices ||
typeof navigator.mediaDevices.getUserMedia !== 'function'
) {
throw new Error('Microphone capture is not supported in this environment');
}
const AudioContextCtor = getAudioContextCtor();
if (!AudioContextCtor) {
throw new Error('AudioContext unavailable');
}
const stream = await navigator.mediaDevices.getUserMedia({
audio: {
channelCount: 1,
noiseSuppression: true,
echoCancellation: true,
autoGainControl: true,
},
});
const context = new AudioContextCtor();
try {
if (context.state === 'suspended') {
await context.resume().catch(() => undefined);
}
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(4096, 1, 1);
const gain = context.createGain();
gain.gain.value = 0;
graphRef.current = {
stream,
context,
source,
processor,
gain,
pending: new Int16Array(0),
started: true,
};
processor.onaudioprocess = (event) => {
const graph = graphRef.current;
if (!graph.started) {
return;
}
const input = event.inputBuffer.getChannelData(0);
let sumSquares = 0;
for (let i = 0; i < input.length; i++) {
sumSquares += input[i] * input[i];
}
const rms = Math.sqrt(sumSquares / Math.max(1, input.length));
setVolume(Math.min(1, Math.max(0, rms * 2)));
const next = resampleToPcm16(input, context.sampleRate, OUTPUT_RATE);
graph.pending = concatInt16(graph.pending, next);
while (graph.pending.length >= CHUNK_SAMPLES) {
const chunk = graph.pending.slice(0, CHUNK_SAMPLES);
graph.pending = graph.pending.slice(CHUNK_SAMPLES);
onPcmSegmentRef.current(int16ToBase64(chunk));
}
};
source.connect(processor);
processor.connect(gain);
gain.connect(context.destination);
} catch (error) {
stream.getTracks().forEach((track) => {
try {
track.stop();
} catch {
// no-op
}
});
try {
await context.close();
} catch {
// no-op
}
graphRef.current = emptyGraph();
throw error instanceof Error ? error : new Error(String(error));
}
}, []);
const stop = useCallback(async () => {
const graph = graphRef.current;
graph.started = false;
setVolume(0);
if (graph.processor) {
try {
graph.processor.onaudioprocess = null;
} catch {
// no-op
}
}
safeDisconnect(graph.processor);
safeDisconnect(graph.source);
safeDisconnect(graph.gain);
if (graph.stream) {
graph.stream.getTracks().forEach((track) => {
try {
track.stop();
} catch {
// no-op
}
});
}
const pending = graph.pending;
graph.pending = new Int16Array(0);
if (pending.length > 0) {
onPcmSegmentRef.current(int16ToBase64(pending));
}
if (graph.context) {
try {
await graph.context.close();
} catch {
// no-op
}
}
// A new capture may have started while the old context was closing;
// only clear the ref if it still points at the graph we tore down.
if (graphRef.current === graph) {
graphRef.current = emptyGraph();
}
}, []);
useEffect(() => {
return () => {
void stop().catch((err) => {
onErrorRef.current?.(err instanceof Error ? err : new Error(String(err)));
});
};
}, [stop]);
return useMemo(
() => ({
start: async () => {
try {
await start();
} catch (err) {
const normalized = err instanceof Error ? err : new Error(String(err));
onErrorRef.current?.(normalized);
throw normalized;
}
},
stop,
volume,
}),
[start, stop, volume],
);
}
packages/ui/src/lib/i18n/messages/en.settings.ts
+20 -3
View File
@@ -1518,16 +1518,33 @@ export const settingsDict = {
'settings.notifications.page.toast.backgroundDisabled': 'Background notifications disabled',
'settings.notifications.page.testNotification.title': 'Test Notification',
'settings.notifications.page.testNotification.body': 'This is a test notification from OpenChamber.',
'settings.voice.page.section.voiceSetup': 'Voice Setup',
'settings.voice.page.section.speechRecognition': 'Speech Recognition',
'settings.voice.page.field.enableVoiceInput': 'Enable voice input',
'settings.voice.page.field.enableVoiceInputAria': 'Enable voice input (dictation)',
'settings.voice.page.section.playbackAndSummary': 'Playback',
'settings.voice.page.field.enableVoiceModeAria': 'Enable voice mode',
'settings.voice.page.field.enableVoiceMode': 'Enable Voice Mode',
'settings.voice.page.field.provider': 'Provider',
'settings.voice.page.provider.browser': 'Browser',
'settings.voice.page.provider.custom': 'Custom',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': 'Server',
'settings.voice.page.provider.local': 'Local',
'settings.voice.page.tooltip.sttLocal': 'On-device transcription on the OpenChamber server. Models download automatically; no API key needed.',
'settings.voice.page.tooltip.localTts': 'On-device synthesis on the OpenChamber server (Kokoro, English). The model downloads automatically; no API key needed.',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2 (English)',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3 (25 European languages)',
'settings.voice.page.stt.model.whisperBase': 'Whisper base (multilingual)',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny (multilingual)',
'settings.voice.page.stt.badge.bestForEnglish': 'Best for English',
'settings.voice.page.stt.badge.bestForMultilingual': 'Best for multilingual',
'settings.voice.page.stt.meta.accuracy': 'Accuracy',
'settings.voice.page.stt.meta.speed': 'Speed',
'settings.voice.page.stt.modelInstalled': 'Model installed',
'settings.voice.page.stt.modelDownloading': 'Downloading model...',
'settings.voice.page.stt.modelDownloadingProgress': 'Downloading model... {percent}%',
'settings.voice.page.stt.modelNotInstalled': 'Model not downloaded',
'settings.voice.page.stt.modelDownload': 'Download',
'settings.voice.page.stt.modelDelete': 'Delete model',
'settings.voice.page.stt.modelRetry': 'Retry',
'settings.voice.page.provider.wasm': 'Local',
'settings.voice.page.stt.wasmModel': 'Whisper Model',
'settings.voice.page.stt.wasmLoaded': 'Model loaded, ready',
packages/ui/src/lib/i18n/messages/en.ts
+12
View File
@@ -1873,6 +1873,18 @@ export const dict = {
'chat.chatInput.placeholder.chat': '@ for files/agents; / for commands and skills; ! for shell; # for snippets',
'chat.chatInput.placeholder.chatCompact': 'Use @ / ! # for helpers',
'chat.chatInput.placeholder.selectSession': 'Select or create a session to start chatting',
'chat.dictation.start': 'Start dictation',
'chat.dictation.overlayAria': 'Dictation',
'chat.dictation.downloadingModel': 'Downloading speech model...',
'chat.dictation.downloadingModelProgress': 'Downloading speech model... {percent}%',
'chat.dictation.listening': 'Listening...',
'chat.dictation.processing': 'Transcribing...',
'chat.dictation.failed': 'Transcription failed',
'chat.dictation.cancel': 'Discard dictation',
'chat.dictation.insert': 'Insert transcript',
'chat.dictation.insertAndSend': 'Insert and send',
'chat.dictation.retry': 'Retry transcription',
'chat.dictation.discard': 'Discard recording',
'chat.snippetAutocomplete.action.addNew': '+ Add new snippet',
'chat.snippetAutocomplete.empty': 'No snippets found',
'chat.snippetAutocomplete.footer': '↑↓ navigate • Enter select • Esc close',
packages/ui/src/lib/i18n/messages/es.settings.ts
+20 -3
View File
@@ -1485,16 +1485,33 @@ export const settingsDict = {
"settings.notifications.page.toast.backgroundDisabled": "Notificaciones de fondo deshabilitadas",
"settings.notifications.page.testNotification.title": "Notificación de prueba",
"settings.notifications.page.testNotification.body": "Esta es una notificación de prueba de OpenChamber.",
"settings.voice.page.section.voiceSetup": "Configuración de voz",
"settings.voice.page.section.speechRecognition": "Reconocimiento de voz",
"settings.voice.page.field.enableVoiceInput": "Habilitar entrada de voz",
"settings.voice.page.field.enableVoiceInputAria": "Habilitar entrada de voz (dictado)",
"settings.voice.page.section.playbackAndSummary": "Reproducción",
"settings.voice.page.field.enableVoiceModeAria": "Habilitar modo de voz",
"settings.voice.page.field.enableVoiceMode": "Habilitar modo de voz",
"settings.voice.page.field.provider": "Proveedor",
"settings.voice.page.provider.browser": "Navegador",
"settings.voice.page.provider.custom": "Personalizado",
"settings.voice.page.provider.say": "Decir",
"settings.voice.page.provider.server": "Servidor",
"settings.voice.page.provider.local": "Local",
"settings.voice.page.tooltip.sttLocal": "Transcripción local en el servidor de OpenChamber. Los modelos se descargan automáticamente; no se necesita clave de API.",
"settings.voice.page.tooltip.localTts": "Síntesis local en el servidor de OpenChamber (Kokoro, inglés). El modelo se descarga automáticamente; no se necesita clave de API.",
"settings.voice.page.stt.model.parakeetV2": "Parakeet v2 (inglés)",
"settings.voice.page.stt.model.parakeetV3": "Parakeet v3 (25 idiomas europeos)",
"settings.voice.page.stt.model.whisperBase": "Whisper base (multilingüe)",
"settings.voice.page.stt.model.whisperTiny": "Whisper tiny (multilingüe)",
"settings.voice.page.stt.badge.bestForEnglish": "Mejor para inglés",
"settings.voice.page.stt.badge.bestForMultilingual": "Mejor multilingüe",
"settings.voice.page.stt.meta.accuracy": "Precisión",
"settings.voice.page.stt.meta.speed": "Velocidad",
"settings.voice.page.stt.modelInstalled": "Modelo instalado",
"settings.voice.page.stt.modelDownloading": "Descargando modelo...",
"settings.voice.page.stt.modelDownloadingProgress": "Descargando modelo... {percent}%",
"settings.voice.page.stt.modelNotInstalled": "Modelo no descargado",
"settings.voice.page.stt.modelDownload": "Descargar",
"settings.voice.page.stt.modelDelete": "Eliminar modelo",
"settings.voice.page.stt.modelRetry": "Reintentar",
"settings.voice.page.provider.wasm": "Local",
"settings.voice.page.stt.wasmModel": "Modelo Whisper",
"settings.voice.page.stt.wasmLoaded": "Modelo cargado",
packages/ui/src/lib/i18n/messages/es.ts
+12
View File
@@ -1245,6 +1245,18 @@ export const dict: Record<I18nKey, string> = {
'diffView.reviewDialog.actions.starting': 'Iniciando...',
'diffView.reviewDialog.toast.noSessionDirectory': 'El directorio de la sesión no está disponible',
'diffView.reviewDialog.toast.startFailed': 'No se pudo iniciar el flujo de revisión',
'chat.dictation.start': 'Iniciar dictado',
'chat.dictation.overlayAria': 'Dictado',
'chat.dictation.downloadingModel': 'Descargando el modelo de voz...',
'chat.dictation.downloadingModelProgress': 'Descargando el modelo de voz... {percent}%',
'chat.dictation.listening': 'Escuchando...',
'chat.dictation.processing': 'Transcribiendo...',
'chat.dictation.failed': 'La transcripción falló',
'chat.dictation.cancel': 'Descartar dictado',
'chat.dictation.insert': 'Insertar transcripción',
'chat.dictation.insertAndSend': 'Insertar y enviar',
'chat.dictation.retry': 'Reintentar transcripción',
'chat.dictation.discard': 'Descartar grabación',
'chat.history.loadOlder': 'Cargar mensajes anteriores',
'chat.autoReview.title': 'El ciclo de revisión de código está en curso',
'chat.autoReview.status.waitingForReviewer': 'Esperando al revisor',
packages/ui/src/lib/i18n/messages/fr.settings.ts
+20 -3
View File
@@ -1468,16 +1468,33 @@ export const settingsDict = {
'settings.notifications.page.toast.backgroundDisabled': 'Notifications en arrière-plan désactivées',
'settings.notifications.page.testNotification.title': 'Avis de test',
'settings.notifications.page.testNotification.body': 'Il s\'agit d\'une notification de test de OpenChamber.',
'settings.voice.page.section.voiceSetup': 'Configuration vocale',
'settings.voice.page.section.speechRecognition': 'Reconnaissance vocale',
'settings.voice.page.field.enableVoiceInput': 'Activer la saisie vocale',
'settings.voice.page.field.enableVoiceInputAria': 'Activer la saisie vocale (dictée)',
'settings.voice.page.section.playbackAndSummary': 'Lecture',
'settings.voice.page.field.enableVoiceModeAria': 'Activer le mode vocal',
'settings.voice.page.field.enableVoiceMode': 'Activer le mode vocal',
'settings.voice.page.field.provider': 'Fournisseur',
'settings.voice.page.provider.browser': 'Navigateur',
'settings.voice.page.provider.custom': 'Personnalisé',
'settings.voice.page.provider.say': 'Dire',
'settings.voice.page.provider.server': 'Serveur',
'settings.voice.page.provider.local': 'Local',
'settings.voice.page.tooltip.sttLocal': 'Transcription locale sur le serveur OpenChamber. Les modèles se téléchargent automatiquement ; aucune clé d\'API requise.',
'settings.voice.page.tooltip.localTts': 'Synthèse locale sur le serveur OpenChamber (Kokoro, anglais). Le modèle se télécharge automatiquement ; aucune clé d’API requise.',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2 (anglais)',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3 (25 langues européennes)',
'settings.voice.page.stt.model.whisperBase': 'Whisper base (multilingue)',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny (multilingue)',
'settings.voice.page.stt.badge.bestForEnglish': 'Idéal pour l’anglais',
'settings.voice.page.stt.badge.bestForMultilingual': 'Idéal multilingue',
'settings.voice.page.stt.meta.accuracy': 'Précision',
'settings.voice.page.stt.meta.speed': 'Vitesse',
'settings.voice.page.stt.modelInstalled': 'Modèle installé',
'settings.voice.page.stt.modelDownloading': 'Téléchargement du modèle...',
'settings.voice.page.stt.modelDownloadingProgress': 'Téléchargement du modèle... {percent}%',
'settings.voice.page.stt.modelNotInstalled': 'Modèle non téléchargé',
'settings.voice.page.stt.modelDownload': 'Télécharger',
'settings.voice.page.stt.modelDelete': 'Supprimer le modèle',
'settings.voice.page.stt.modelRetry': 'Réessayer',
'settings.voice.page.provider.wasm': 'Local',
'settings.voice.page.stt.wasmModel': 'Modèle Whisper',
'settings.voice.page.stt.wasmLoaded': 'Modèle chargé, prêt',
packages/ui/src/lib/i18n/messages/fr.ts
+12
View File
@@ -1669,6 +1669,18 @@ export const dict = {
'chat.chatInput.placeholder.chat': '@ pour les fichiers/agents ; / pour les commandes et les skills ; ! pour shell ; # pour les extraits',
'chat.chatInput.placeholder.chatCompact': 'Utiliser @ / ! # pour les aides',
'chat.chatInput.placeholder.selectSession': 'Sélectionnez ou créez une session pour commencer à discuter',
'chat.dictation.start': 'Démarrer la dictée',
'chat.dictation.overlayAria': 'Dictée',
'chat.dictation.downloadingModel': 'Téléchargement du modèle vocal...',
'chat.dictation.downloadingModelProgress': 'Téléchargement du modèle vocal... {percent}%',
'chat.dictation.listening': 'Écoute...',
'chat.dictation.processing': 'Transcription...',
'chat.dictation.failed': 'Échec de la transcription',
'chat.dictation.cancel': 'Abandonner la dictée',
'chat.dictation.insert': 'Insérer la transcription',
'chat.dictation.insertAndSend': 'Insérer et envoyer',
'chat.dictation.retry': 'Réessayer la transcription',
'chat.dictation.discard': 'Abandonner l\'enregistrement',
'chat.snippetAutocomplete.action.addNew': '+ Ajouter un nouvel extrait',
'chat.snippetAutocomplete.empty': 'Aucun extrait trouvé',
'chat.snippetAutocomplete.footer': '↑↓ naviguer • Entrer sélectionner • Esc fermer',
packages/ui/src/lib/i18n/messages/ja.settings.ts
+20 -3
View File
@@ -1518,16 +1518,33 @@ export const settingsDict = {
'settings.notifications.page.toast.backgroundDisabled': 'バックグラウンド通知を無効化しました',
'settings.notifications.page.testNotification.title': 'テスト通知',
'settings.notifications.page.testNotification.body': 'これは OpenChamber からのテスト通知です。',
'settings.voice.page.section.voiceSetup': '音声設定',
'settings.voice.page.section.speechRecognition': '音声認識',
'settings.voice.page.field.enableVoiceInput': '音声入力を有効にする',
'settings.voice.page.field.enableVoiceInputAria': '音声入力（ディクテーション）を有効にする',
'settings.voice.page.section.playbackAndSummary': '再生',
'settings.voice.page.field.enableVoiceModeAria': '音声モードを有効化',
'settings.voice.page.field.enableVoiceMode': '音声モードを有効化',
'settings.voice.page.field.provider': 'プロバイダー',
'settings.voice.page.provider.browser': 'ブラウザ',
'settings.voice.page.provider.custom': 'カスタム',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': 'サーバー',
'settings.voice.page.provider.local': 'ローカル',
'settings.voice.page.tooltip.sttLocal': 'OpenChamber サーバー上でローカルに文字起こしします。モデルは自動でダウンロードされ、API キーは不要です。',
'settings.voice.page.tooltip.localTts': 'OpenChamber サーバー上でローカルに音声合成します（Kokoro、英語）。モデルは自動でダウンロードされ、API キーは不要です。',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2（英語）',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3（ヨーロッパ25言語）',
'settings.voice.page.stt.model.whisperBase': 'Whisper base（多言語）',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny（多言語）',
'settings.voice.page.stt.badge.bestForEnglish': '英語に最適',
'settings.voice.page.stt.badge.bestForMultilingual': '多言語に最適',
'settings.voice.page.stt.meta.accuracy': '精度',
'settings.voice.page.stt.meta.speed': '速度',
'settings.voice.page.stt.modelInstalled': 'モデルはインストール済み',
'settings.voice.page.stt.modelDownloading': 'モデルをダウンロード中...',
'settings.voice.page.stt.modelDownloadingProgress': 'モデルをダウンロード中... {percent}%',
'settings.voice.page.stt.modelNotInstalled': 'モデル未ダウンロード',
'settings.voice.page.stt.modelDownload': 'ダウンロード',
'settings.voice.page.stt.modelDelete': 'モデルを削除',
'settings.voice.page.stt.modelRetry': '再試行',
'settings.voice.page.provider.wasm': 'ローカル',
'settings.voice.page.stt.wasmModel': 'Whisper モデル',
'settings.voice.page.stt.wasmLoaded': 'モデル読み込み完了',
packages/ui/src/lib/i18n/messages/ja.ts
+12
View File
@@ -1869,6 +1869,18 @@ export const dict: Record<I18nKey, string> = {
'chat.chatInput.placeholder.chat': '@でファイル/エージェント、/でコマンド/スキル、!でシェル、#でスニペット',
'chat.chatInput.placeholder.chatCompact': '@ / ! # でヘルパーを使用',
'chat.chatInput.placeholder.selectSession': 'セッションを選択または作成してチャットを開始',
'chat.dictation.start': '音声入力を開始',
'chat.dictation.overlayAria': '音声入力',
'chat.dictation.downloadingModel': '音声モデルをダウンロード中...',
'chat.dictation.downloadingModelProgress': '音声モデルをダウンロード中... {percent}%',
'chat.dictation.listening': '聞き取り中...',
'chat.dictation.processing': '文字起こし中...',
'chat.dictation.failed': '文字起こしに失敗しました',
'chat.dictation.cancel': '音声入力を破棄',
'chat.dictation.insert': 'テキストを挿入',
'chat.dictation.insertAndSend': '挿入して送信',
'chat.dictation.retry': '文字起こしを再試行',
'chat.dictation.discard': '録音を破棄',
'chat.snippetAutocomplete.action.addNew': '+ 新しいスニペットを追加',
'chat.snippetAutocomplete.empty': 'スニペットが見つかりません',
'chat.snippetAutocomplete.footer': '↑↓ 移動 • Enter 選択 • Esc 閉じる',
packages/ui/src/lib/i18n/messages/ko.settings.ts
+20 -3
View File
@@ -1485,16 +1485,33 @@ export const settingsDict = {
'settings.notifications.page.toast.backgroundDisabled': '백그라운드 알림이 비활성화되었습니다',
'settings.notifications.page.testNotification.title': '테스트 알림',
'settings.notifications.page.testNotification.body': 'OpenChamber의 테스트 알림입니다.',
'settings.voice.page.section.voiceSetup': '음성',
'settings.voice.page.section.speechRecognition': '음성 인식',
'settings.voice.page.field.enableVoiceInput': '음성 입력 사용',
'settings.voice.page.field.enableVoiceInputAria': '음성 입력(받아쓰기) 사용',
'settings.voice.page.section.playbackAndSummary': '재생',
'settings.voice.page.field.enableVoiceModeAria': '음성 모드 활성화',
'settings.voice.page.field.enableVoiceMode': '음성 모드 활성화',
'settings.voice.page.field.provider': '프로바이더',
'settings.voice.page.provider.browser': '브라우저',
'settings.voice.page.provider.custom': '사용자 정의',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': '서버',
'settings.voice.page.provider.local': '로컬',
'settings.voice.page.tooltip.sttLocal': 'OpenChamber 서버에서 로컬로 변환합니다. 모델은 자동으로 다운로드되며 API 키가 필요 없습니다.',
'settings.voice.page.tooltip.localTts': 'OpenChamber 서버에서 로컬로 음성을 합성합니다(Kokoro, 영어). 모델은 자동으로 다운로드되며 API 키가 필요 없습니다.',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2 (영어)',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3 (유럽 25개 언어)',
'settings.voice.page.stt.model.whisperBase': 'Whisper base (다국어)',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny (다국어)',
'settings.voice.page.stt.badge.bestForEnglish': '영어에 최적',
'settings.voice.page.stt.badge.bestForMultilingual': '다국어에 최적',
'settings.voice.page.stt.meta.accuracy': '정확도',
'settings.voice.page.stt.meta.speed': '속도',
'settings.voice.page.stt.modelInstalled': '모델 설치됨',
'settings.voice.page.stt.modelDownloading': '모델 다운로드 중...',
'settings.voice.page.stt.modelDownloadingProgress': '모델 다운로드 중... {percent}%',
'settings.voice.page.stt.modelNotInstalled': '모델이 다운로드되지 않음',
'settings.voice.page.stt.modelDownload': '다운로드',
'settings.voice.page.stt.modelDelete': '모델 삭제',
'settings.voice.page.stt.modelRetry': '다시 시도',
'settings.voice.page.provider.wasm': '로컬',
'settings.voice.page.stt.wasmModel': 'Whisper 모델',
'settings.voice.page.stt.wasmLoaded': '모델 로드됨',
packages/ui/src/lib/i18n/messages/ko.ts
+12
View File
@@ -1873,6 +1873,18 @@ export const dict: Record<I18nKey, string> = {
'chat.chatInput.placeholder.chat': '@ 파일/에이전트; / 명령 및 스킬; ! shell; # 스니펫',
'chat.chatInput.placeholder.chatCompact': '@ / ! # 도우미 사용',
'chat.chatInput.placeholder.selectSession': '채팅을 시작할 세션을 선택하거나 새로 만드세요',
'chat.dictation.start': '받아쓰기 시작',
'chat.dictation.overlayAria': '받아쓰기',
'chat.dictation.downloadingModel': '음성 모델 다운로드 중...',
'chat.dictation.downloadingModelProgress': '음성 모델 다운로드 중... {percent}%',
'chat.dictation.listening': '듣는 중...',
'chat.dictation.processing': '변환 중...',
'chat.dictation.failed': '변환에 실패했습니다',
'chat.dictation.cancel': '받아쓰기 취소',
'chat.dictation.insert': '텍스트 삽입',
'chat.dictation.insertAndSend': '삽입 후 전송',
'chat.dictation.retry': '변환 다시 시도',
'chat.dictation.discard': '녹음 삭제',
'chat.snippetAutocomplete.action.addNew': '+ 새 스니펫 추가',
'chat.snippetAutocomplete.empty': '스니펫을 찾을 수 없음',
'chat.snippetAutocomplete.footer': '↑↓ 이동 • Enter 선택 • Esc 닫기',
packages/ui/src/lib/i18n/messages/pl.settings.ts
+20 -3
View File
@@ -1743,8 +1743,6 @@ export const settingsDict = {
'settings.voice.page.field.apiKeyHintUsingConfig': 'Użyto klucza z konfiguracji',
'settings.voice.page.field.auto': 'Auto',
'settings.voice.page.field.configuredAbove': 'Skonfigurowano powyżej',
'settings.voice.page.field.enableVoiceMode': 'Włącz tryb głosowy',
'settings.voice.page.field.enableVoiceModeAria': 'Włącz tryb głosowy',
'settings.voice.page.field.language': 'Język',
'settings.voice.page.field.messageReadAloudButton': 'Przycisk czytania wiadomości na głos',
'settings.voice.page.field.messageReadAloudButtonAria': 'Przycisk czytania wiadomości na głos',
@@ -1782,6 +1780,24 @@ export const settingsDict = {
'settings.voice.page.provider.custom': 'Własny',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': 'Serwer',
'settings.voice.page.provider.local': 'Lokalny',
'settings.voice.page.tooltip.sttLocal': 'Transkrypcja lokalna na serwerze OpenChamber. Modele pobierają się automatycznie; klucz API nie jest potrzebny.',
'settings.voice.page.tooltip.localTts': 'Lokalna synteza na serwerze OpenChamber (Kokoro, angielski). Model pobiera się automatycznie; klucz API nie jest potrzebny.',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2 (angielski)',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3 (25 języków europejskich)',
'settings.voice.page.stt.model.whisperBase': 'Whisper base (wielojęzyczny)',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny (wielojęzyczny)',
'settings.voice.page.stt.badge.bestForEnglish': 'Najlepszy do angielskiego',
'settings.voice.page.stt.badge.bestForMultilingual': 'Najlepszy wielojęzyczny',
'settings.voice.page.stt.meta.accuracy': 'Dokładność',
'settings.voice.page.stt.meta.speed': 'Szybkość',
'settings.voice.page.stt.modelInstalled': 'Model zainstalowany',
'settings.voice.page.stt.modelDownloading': 'Pobieranie modelu...',
'settings.voice.page.stt.modelDownloadingProgress': 'Pobieranie modelu... {percent}%',
'settings.voice.page.stt.modelNotInstalled': 'Model niepobrany',
'settings.voice.page.stt.modelDownload': 'Pobierz',
'settings.voice.page.stt.modelDelete': 'Usuń model',
'settings.voice.page.stt.modelRetry': 'Ponów',
'settings.voice.page.provider.wasm': 'Lokalny',
'settings.voice.page.stt.wasmModel': 'Model Whisper',
'settings.voice.page.stt.wasmLoaded': 'Model załadowany',
@@ -1792,7 +1808,8 @@ export const settingsDict = {
'settings.voice.page.stt.wasmRetry': 'Spróbuj ponownie',
'settings.voice.page.section.playbackAndSummary': 'Odtwarzanie',
'settings.voice.page.section.speechRecognition': 'Rozpoznawanie mowy',
'settings.voice.page.section.voiceSetup': 'Konfiguracja głosu',
'settings.voice.page.field.enableVoiceInput': 'Włącz wprowadzanie głosowe',
'settings.voice.page.field.enableVoiceInputAria': 'Włącz wprowadzanie głosowe (dyktowanie)',
'settings.voice.page.tooltip.browser': 'Darmowe, offline, ograniczone wsparcie mobilne.',
'settings.voice.page.tooltip.custom': 'Serwer zgodny z OpenAI (na przykład Kokoro).',
'settings.voice.page.tooltip.openai': 'Wysoka jakość, gotowe na urządzenia mobilne, wymaga klucza API.',
packages/ui/src/lib/i18n/messages/pl.ts
+12
View File
@@ -1091,6 +1091,18 @@ export const dict: Record<I18nKey, string> = {
'chat.chatInput.placeholder.chatCompact': 'Użyj @ / ! # dla pomocników',
'chat.chatInput.placeholder.selectSession': 'Wybierz lub utwórz sesję, aby zacząć czatować',
'chat.chatInput.placeholder.shell': 'Wpisz polecenie powłoki...',
'chat.dictation.start': 'Rozpocznij dyktowanie',
'chat.dictation.overlayAria': 'Dyktowanie',
'chat.dictation.downloadingModel': 'Pobieranie modelu mowy...',
'chat.dictation.downloadingModelProgress': 'Pobieranie modelu mowy... {percent}%',
'chat.dictation.listening': 'Słucham...',
'chat.dictation.processing': 'Transkrybowanie...',
'chat.dictation.failed': 'Transkrypcja nie powiodła się',
'chat.dictation.cancel': 'Odrzuć dyktowanie',
'chat.dictation.insert': 'Wstaw transkrypcję',
'chat.dictation.insertAndSend': 'Wstaw i wyślij',
'chat.dictation.retry': 'Ponów transkrypcję',
'chat.dictation.discard': 'Odrzuć nagranie',
'chat.snippetAutocomplete.action.addNew': '+ Dodaj nowy fragment',
'chat.snippetAutocomplete.empty': 'Nie znaleziono fragmentów',
'chat.snippetAutocomplete.footer': '↑↓ nawigacja • Enter wybierz • Esc zamknij',
packages/ui/src/lib/i18n/messages/pt-BR.settings.ts
+20 -3
View File
@@ -1485,16 +1485,33 @@ export const settingsDict = {
"settings.notifications.page.toast.backgroundDisabled": "Notificações em segundo plano desabilitadas",
"settings.notifications.page.testNotification.title": "Notificação de teste",
"settings.notifications.page.testNotification.body": "Esta é uma notificação de teste do OpenChamber.",
"settings.voice.page.section.voiceSetup": "Configurações de voz",
"settings.voice.page.section.speechRecognition": "Reconhecimento de voz",
"settings.voice.page.field.enableVoiceInput": "Ativar entrada por voz",
"settings.voice.page.field.enableVoiceInputAria": "Ativar entrada por voz (ditado)",
"settings.voice.page.section.playbackAndSummary": "Reprodução",
"settings.voice.page.field.enableVoiceModeAria": "Ativar modo de voz",
"settings.voice.page.field.enableVoiceMode": "Ativar modo de voz",
"settings.voice.page.field.provider": "Provedor",
"settings.voice.page.provider.browser": "Navegador",
"settings.voice.page.provider.custom": "Personalizado",
"settings.voice.page.provider.say": "Falar",
"settings.voice.page.provider.server": "Servidor",
"settings.voice.page.provider.local": "Local",
"settings.voice.page.tooltip.sttLocal": "Transcrição local no servidor do OpenChamber. Os modelos são baixados automaticamente; não é necessária chave de API.",
"settings.voice.page.tooltip.localTts": "Síntese local no servidor do OpenChamber (Kokoro, inglês). O modelo é baixado automaticamente; não é necessária chave de API.",
"settings.voice.page.stt.model.parakeetV2": "Parakeet v2 (inglês)",
"settings.voice.page.stt.model.parakeetV3": "Parakeet v3 (25 idiomas europeus)",
"settings.voice.page.stt.model.whisperBase": "Whisper base (multilíngue)",
"settings.voice.page.stt.model.whisperTiny": "Whisper tiny (multilíngue)",
"settings.voice.page.stt.badge.bestForEnglish": "Melhor para inglês",
"settings.voice.page.stt.badge.bestForMultilingual": "Melhor multilíngue",
"settings.voice.page.stt.meta.accuracy": "Precisão",
"settings.voice.page.stt.meta.speed": "Velocidade",
"settings.voice.page.stt.modelInstalled": "Modelo instalado",
"settings.voice.page.stt.modelDownloading": "Baixando modelo...",
"settings.voice.page.stt.modelDownloadingProgress": "Baixando modelo... {percent}%",
"settings.voice.page.stt.modelNotInstalled": "Modelo não baixado",
"settings.voice.page.stt.modelDownload": "Baixar",
"settings.voice.page.stt.modelDelete": "Excluir modelo",
"settings.voice.page.stt.modelRetry": "Tentar novamente",
"settings.voice.page.provider.wasm": "Local",
"settings.voice.page.stt.wasmModel": "Modelo Whisper",
"settings.voice.page.stt.wasmLoaded": "Modelo carregado",
packages/ui/src/lib/i18n/messages/pt-BR.ts
+12
View File
@@ -1245,6 +1245,18 @@ export const dict: Record<I18nKey, string> = {
'diffView.reviewDialog.actions.starting': 'Iniciando...',
'diffView.reviewDialog.toast.noSessionDirectory': 'O diretório da sessão está indisponível',
'diffView.reviewDialog.toast.startFailed': 'Falha ao iniciar o fluxo de revisão',
'chat.dictation.start': 'Iniciar ditado',
'chat.dictation.overlayAria': 'Ditado',
'chat.dictation.downloadingModel': 'Baixando o modelo de fala...',
'chat.dictation.downloadingModelProgress': 'Baixando o modelo de fala... {percent}%',
'chat.dictation.listening': 'Ouvindo...',
'chat.dictation.processing': 'Transcrevendo...',
'chat.dictation.failed': 'Falha na transcrição',
'chat.dictation.cancel': 'Descartar ditado',
'chat.dictation.insert': 'Inserir transcrição',
'chat.dictation.insertAndSend': 'Inserir e enviar',
'chat.dictation.retry': 'Tentar transcrever novamente',
'chat.dictation.discard': 'Descartar gravação',
'chat.history.loadOlder': 'Carregar mensagens anteriores',
'chat.autoReview.title': 'O ciclo de revisão de código está em andamento',
'chat.autoReview.status.waitingForReviewer': 'Aguardando o revisor',
packages/ui/src/lib/i18n/messages/uk.settings.ts
+20 -3
View File
@@ -1485,16 +1485,33 @@ export const settingsDict = {
"settings.notifications.page.toast.backgroundDisabled": "Фонові сповіщення вимкнено",
"settings.notifications.page.testNotification.title": "Тестове сповіщення",
"settings.notifications.page.testNotification.body": "Це тестове сповіщення від OpenChamber.",
"settings.voice.page.section.voiceSetup": "Налаштування голосу",
"settings.voice.page.section.speechRecognition": "Розпізнавання мовлення",
"settings.voice.page.field.enableVoiceInput": "Увімкнути голосовий ввід",
"settings.voice.page.field.enableVoiceInputAria": "Увімкнути голосовий ввід (диктування)",
"settings.voice.page.section.playbackAndSummary": "Відтворення",
"settings.voice.page.field.enableVoiceModeAria": "Увімкнути голосовий режим",
"settings.voice.page.field.enableVoiceMode": "Увімкнути голосовий режим",
"settings.voice.page.field.provider": "Провайдер",
"settings.voice.page.provider.browser": "Браузер",
"settings.voice.page.provider.custom": "Власний",
"settings.voice.page.provider.say": "Say",
"settings.voice.page.provider.server": "Сервер",
"settings.voice.page.provider.local": "Локальний",
"settings.voice.page.tooltip.sttLocal": "Локальна розшифровка на сервері OpenChamber. Моделі завантажуються автоматично; ключ API не потрібен.",
"settings.voice.page.tooltip.localTts": "Локальний синтез на сервері OpenChamber (Kokoro, англійська). Модель завантажується автоматично; ключ API не потрібен.",
"settings.voice.page.stt.model.parakeetV2": "Parakeet v2 (англійська)",
"settings.voice.page.stt.model.parakeetV3": "Parakeet v3 (25 європейських мов)",
"settings.voice.page.stt.model.whisperBase": "Whisper base (мультимовна)",
"settings.voice.page.stt.model.whisperTiny": "Whisper tiny (мультимовна)",
"settings.voice.page.stt.badge.bestForEnglish": "Найкраща для англійської",
"settings.voice.page.stt.badge.bestForMultilingual": "Найкраща мультимовна",
"settings.voice.page.stt.meta.accuracy": "Точність",
"settings.voice.page.stt.meta.speed": "Швидкість",
"settings.voice.page.stt.modelInstalled": "Модель встановлено",
"settings.voice.page.stt.modelDownloading": "Завантаження моделі...",
"settings.voice.page.stt.modelDownloadingProgress": "Завантаження моделі... {percent}%",
"settings.voice.page.stt.modelNotInstalled": "Модель не завантажено",
"settings.voice.page.stt.modelDownload": "Завантажити",
"settings.voice.page.stt.modelDelete": "Видалити модель",
"settings.voice.page.stt.modelRetry": "Повторити",
"settings.voice.page.provider.wasm": "Локально",
"settings.voice.page.stt.wasmModel": "Модель Whisper",
"settings.voice.page.stt.wasmLoaded": "Модель завантажено",
packages/ui/src/lib/i18n/messages/uk.ts
+12
View File
@@ -1245,6 +1245,18 @@ export const dict: Record<I18nKey, string> = {
'diffView.reviewDialog.actions.starting': 'Запуск...',
'diffView.reviewDialog.toast.noSessionDirectory': 'Директорія сесії недоступна',
'diffView.reviewDialog.toast.startFailed': 'Не вдалося запустити review flow',
'chat.dictation.start': 'Почати диктування',
'chat.dictation.overlayAria': 'Диктування',
'chat.dictation.downloadingModel': 'Завантаження мовної моделі...',
'chat.dictation.downloadingModelProgress': 'Завантаження мовної моделі... {percent}%',
'chat.dictation.listening': 'Слухаю...',
'chat.dictation.processing': 'Розшифровка...',
'chat.dictation.failed': 'Не вдалося розшифрувати запис',
'chat.dictation.cancel': 'Відхилити диктування',
'chat.dictation.insert': 'Вставити текст',
'chat.dictation.insertAndSend': 'Вставити й надіслати',
'chat.dictation.retry': 'Повторити розшифровку',
'chat.dictation.discard': 'Відхилити запис',
'chat.history.loadOlder': 'Завантажити ще',
'chat.autoReview.title': 'Цикл код-ревʼю триває',
'chat.autoReview.status.waitingForReviewer': 'Очікуємо ревʼювера',
packages/ui/src/lib/i18n/messages/zh-CN.settings.ts
+20 -3
View File
@@ -1485,16 +1485,33 @@ export const settingsDict = {
'settings.notifications.page.toast.backgroundDisabled': '后台通知已关闭',
'settings.notifications.page.testNotification.title': '测试通知',
'settings.notifications.page.testNotification.body': '这是来自 OpenChamber 的测试通知。',
'settings.voice.page.section.voiceSetup': '语音设置',
'settings.voice.page.section.speechRecognition': '语音识别',
'settings.voice.page.field.enableVoiceInput': '启用语音输入',
'settings.voice.page.field.enableVoiceInputAria': '启用语音输入（听写）',
'settings.voice.page.section.playbackAndSummary': '播放',
'settings.voice.page.field.enableVoiceModeAria': '启用语音模式',
'settings.voice.page.field.enableVoiceMode': '启用语音模式',
'settings.voice.page.field.provider': '提供方',
'settings.voice.page.provider.browser': '浏览器',
'settings.voice.page.provider.custom': '自定义',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': '服务器',
'settings.voice.page.provider.local': '本地',
'settings.voice.page.tooltip.sttLocal': '在 OpenChamber 服务器上本地转写。模型自动下载，无需 API 密钥。',
'settings.voice.page.tooltip.localTts': '在 OpenChamber 服务器上本地合成语音（Kokoro，英语）。模型自动下载，无需 API 密钥。',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2（英语）',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3（25 种欧洲语言）',
'settings.voice.page.stt.model.whisperBase': 'Whisper base（多语言）',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny（多语言）',
'settings.voice.page.stt.badge.bestForEnglish': '英语最佳',
'settings.voice.page.stt.badge.bestForMultilingual': '多语言最佳',
'settings.voice.page.stt.meta.accuracy': '准确度',
'settings.voice.page.stt.meta.speed': '速度',
'settings.voice.page.stt.modelInstalled': '模型已安装',
'settings.voice.page.stt.modelDownloading': '正在下载模型...',
'settings.voice.page.stt.modelDownloadingProgress': '正在下载模型... {percent}%',
'settings.voice.page.stt.modelNotInstalled': '模型未下载',
'settings.voice.page.stt.modelDownload': '下载',
'settings.voice.page.stt.modelDelete': '删除模型',
'settings.voice.page.stt.modelRetry': '重试',
'settings.voice.page.provider.wasm': '本地',
'settings.voice.page.stt.wasmModel': 'Whisper 模型',
'settings.voice.page.stt.wasmLoaded': '模型已加载',
packages/ui/src/lib/i18n/messages/zh-CN.ts
+12
View File
@@ -1839,6 +1839,18 @@ export const dict: Record<I18nKey, string> = {
'chat.chatInput.placeholder.chat': '@ 用于文件/智能体；/ 用于命令和技能；! 用于 shell；# 用于代码片段',
'chat.chatInput.placeholder.chatCompact': '使用 @ / ! # 辅助',
'chat.chatInput.placeholder.selectSession': '选择或创建会话以开始聊天',
'chat.dictation.start': '开始语音输入',
'chat.dictation.overlayAria': '语音输入',
'chat.dictation.downloadingModel': '正在下载语音模型...',
'chat.dictation.downloadingModelProgress': '正在下载语音模型... {percent}%',
'chat.dictation.listening': '正在聆听...',
'chat.dictation.processing': '正在转写...',
'chat.dictation.failed': '转写失败',
'chat.dictation.cancel': '放弃语音输入',
'chat.dictation.insert': '插入文本',
'chat.dictation.insertAndSend': '插入并发送',
'chat.dictation.retry': '重试转写',
'chat.dictation.discard': '放弃录音',
'chat.snippetAutocomplete.action.addNew': '+ 新建代码片段',
'chat.snippetAutocomplete.empty': '未找到代码片段',
'chat.snippetAutocomplete.footer': '↑↓ 导航 • Enter 选择 • Esc 关闭',
packages/ui/src/lib/i18n/messages/zh-TW.settings.ts
+20 -3
View File
@@ -1401,16 +1401,33 @@
'settings.notifications.page.toast.backgroundDisabled': '背景通知已關閉',
'settings.notifications.page.testNotification.title': '測試通知',
'settings.notifications.page.testNotification.body': '這是來自 OpenChamber 的測試通知。',
'settings.voice.page.section.voiceSetup': '語音設定',
'settings.voice.page.section.speechRecognition': '語音辨識',
'settings.voice.page.field.enableVoiceInput': '啟用語音輸入',
'settings.voice.page.field.enableVoiceInputAria': '啟用語音輸入（聽寫）',
'settings.voice.page.section.playbackAndSummary': '播放',
'settings.voice.page.field.enableVoiceModeAria': '啟用語音模式',
'settings.voice.page.field.enableVoiceMode': '啟用語音模式',
'settings.voice.page.field.provider': '提供方',
'settings.voice.page.provider.browser': '瀏覽器',
'settings.voice.page.provider.custom': '自訂',
'settings.voice.page.provider.say': 'Say',
'settings.voice.page.provider.server': '伺服器',
'settings.voice.page.provider.local': '本機',
'settings.voice.page.tooltip.sttLocal': '在 OpenChamber 伺服器上本機轉寫。模型會自動下載，無需 API 金鑰。',
'settings.voice.page.tooltip.localTts': '在 OpenChamber 伺服器上本機合成語音（Kokoro，英文）。模型會自動下載，無需 API 金鑰。',
'settings.voice.page.stt.model.parakeetV2': 'Parakeet v2（英文）',
'settings.voice.page.stt.model.parakeetV3': 'Parakeet v3（25 種歐洲語言）',
'settings.voice.page.stt.model.whisperBase': 'Whisper base（多語言）',
'settings.voice.page.stt.model.whisperTiny': 'Whisper tiny（多語言）',
'settings.voice.page.stt.badge.bestForEnglish': '英文最佳',
'settings.voice.page.stt.badge.bestForMultilingual': '多語言最佳',
'settings.voice.page.stt.meta.accuracy': '準確度',
'settings.voice.page.stt.meta.speed': '速度',
'settings.voice.page.stt.modelInstalled': '模型已安裝',
'settings.voice.page.stt.modelDownloading': '正在下載模型...',
'settings.voice.page.stt.modelDownloadingProgress': '正在下載模型... {percent}%',
'settings.voice.page.stt.modelNotInstalled': '模型未下載',
'settings.voice.page.stt.modelDownload': '下載',
'settings.voice.page.stt.modelDelete': '刪除模型',
'settings.voice.page.stt.modelRetry': '重試',
'settings.voice.page.provider.wasm': '本機',
'settings.voice.page.stt.wasmModel': 'Whisper 模型',
'settings.voice.page.stt.wasmLoaded': '模型已載入',
packages/ui/src/lib/i18n/messages/zh-TW.ts
+12
View File
@@ -1843,6 +1843,18 @@ export const dict: Record<I18nKey, string> = {
'chat.chatInput.placeholder.chat': '@ 用於檔案/Agent；/ 用於命令；! 用於 shell；# 用於程式片段',
'chat.chatInput.placeholder.chatCompact': '使用 @ / ! # 輔助',
'chat.chatInput.placeholder.selectSession': '選擇或建立會話以開始聊天',
'chat.dictation.start': '開始語音輸入',
'chat.dictation.overlayAria': '語音輸入',
'chat.dictation.downloadingModel': '正在下載語音模型...',
'chat.dictation.downloadingModelProgress': '正在下載語音模型... {percent}%',
'chat.dictation.listening': '正在聆聽...',
'chat.dictation.processing': '正在轉寫...',
'chat.dictation.failed': '轉寫失敗',
'chat.dictation.cancel': '放棄語音輸入',
'chat.dictation.insert': '插入文字',
'chat.dictation.insertAndSend': '插入並傳送',
'chat.dictation.retry': '重試轉寫',
'chat.dictation.discard': '放棄錄音',
'chat.snippetAutocomplete.action.addNew': '+ 新建程式片段',
'chat.snippetAutocomplete.empty': '未找到程式片段',
'chat.snippetAutocomplete.footer': '↑↓ 導航 • Enter 選擇 • Esc 關閉',
packages/ui/src/lib/persistence.ts
+26 -30
View File
@@ -113,7 +113,10 @@ const persistToLocalStorage = (settings: DesktopSettings) => {
localStorage.removeItem('opencode-update-toast-dismissed-version');
}
}
if (settings.sttProvider === 'browser' || settings.sttProvider === 'server') {
if (typeof settings.dictationEnabled === 'boolean') {
localStorage.setItem('dictationEnabled', String(settings.dictationEnabled));
}
if (settings.sttProvider === 'local' || settings.sttProvider === 'openai-compatible') {
localStorage.setItem('sttProvider', settings.sttProvider);
}
if (typeof settings.sttServerUrl === 'string') {
@@ -122,18 +125,12 @@ const persistToLocalStorage = (settings: DesktopSettings) => {
if (typeof settings.sttModel === 'string') {
localStorage.setItem('sttModel', settings.sttModel);
}
if (typeof settings.sttLocalModel === 'string') {
localStorage.setItem('sttLocalModel', settings.sttLocalModel);
}
if (typeof settings.sttLanguage === 'string') {
localStorage.setItem('sttLanguage', settings.sttLanguage);
}
if (typeof settings.sttSilenceThresholdDb === 'number' && Number.isFinite(settings.sttSilenceThresholdDb)) {
localStorage.setItem('sttSilenceThresholdDb', String(settings.sttSilenceThresholdDb));
}
if (typeof settings.sttSilenceHoldMs === 'number' && Number.isFinite(settings.sttSilenceHoldMs)) {
localStorage.setItem('sttSilenceHoldMs', String(settings.sttSilenceHoldMs));
}
if (typeof settings.sttTranscribeOnStop === 'boolean') {
localStorage.setItem('sttTranscribeOnStop', String(settings.sttTranscribeOnStop));
}
};
const dispatchSettingsSynced = (settings: DesktopSettings): void => {
@@ -614,7 +611,10 @@ const applyDesktopUiPreferences = (settings: DesktopSettings) => {
}
if (configStoreApi && configStore) {
const nextConfigState: Partial<typeof configStore> = {};
if ((settings.sttProvider === 'browser' || settings.sttProvider === 'server') && settings.sttProvider !== configStore.sttProvider) {
if (typeof settings.dictationEnabled === 'boolean' && settings.dictationEnabled !== configStore.dictationEnabled) {
nextConfigState.dictationEnabled = settings.dictationEnabled;
}
if ((settings.sttProvider === 'local' || settings.sttProvider === 'openai-compatible') && settings.sttProvider !== configStore.sttProvider) {
nextConfigState.sttProvider = settings.sttProvider;
}
if (typeof settings.sttServerUrl === 'string' && settings.sttServerUrl !== configStore.sttServerUrl) {
@@ -623,18 +623,12 @@ const applyDesktopUiPreferences = (settings: DesktopSettings) => {
if (typeof settings.sttModel === 'string' && settings.sttModel !== configStore.sttModel) {
nextConfigState.sttModel = settings.sttModel;
}
if (typeof settings.sttLocalModel === 'string' && settings.sttLocalModel !== configStore.sttLocalModel) {
nextConfigState.sttLocalModel = settings.sttLocalModel;
}
if (typeof settings.sttLanguage === 'string' && settings.sttLanguage !== configStore.sttLanguage) {
nextConfigState.sttLanguage = settings.sttLanguage;
}
if (typeof settings.sttSilenceThresholdDb === 'number' && Number.isFinite(settings.sttSilenceThresholdDb) && settings.sttSilenceThresholdDb !== configStore.sttSilenceThresholdDb) {
nextConfigState.sttSilenceThresholdDb = settings.sttSilenceThresholdDb;
}
if (typeof settings.sttSilenceHoldMs === 'number' && Number.isFinite(settings.sttSilenceHoldMs) && settings.sttSilenceHoldMs !== configStore.sttSilenceHoldMs) {
nextConfigState.sttSilenceHoldMs = settings.sttSilenceHoldMs;
}
if (typeof settings.sttTranscribeOnStop === 'boolean' && settings.sttTranscribeOnStop !== configStore.sttTranscribeOnStop) {
nextConfigState.sttTranscribeOnStop = settings.sttTranscribeOnStop;
}
if (Object.keys(nextConfigState).length > 0) {
configStoreApi.setState(nextConfigState);
}
@@ -1200,8 +1194,16 @@ const sanitizeWebSettings = (payload: unknown): DesktopSettings | null => {
if (typeof candidate.responseStyleCustomInstructions === 'string') {
result.responseStyleCustomInstructions = candidate.responseStyleCustomInstructions;
}
if (candidate.sttProvider === 'browser' || candidate.sttProvider === 'server') {
if (typeof candidate.dictationEnabled === 'boolean') {
result.dictationEnabled = candidate.dictationEnabled;
}
if (candidate.sttProvider === 'local' || candidate.sttProvider === 'openai-compatible') {
result.sttProvider = candidate.sttProvider;
} else if (candidate.sttProvider === 'server') {
// Legacy provider migration: 'server' was the OpenAI-compatible endpoint.
result.sttProvider = 'openai-compatible';
} else if (candidate.sttProvider === 'browser' || candidate.sttProvider === 'wasm') {
result.sttProvider = 'local';
}
if (typeof candidate.sttServerUrl === 'string') {
result.sttServerUrl = candidate.sttServerUrl.trim();
@@ -1209,18 +1211,12 @@ const sanitizeWebSettings = (payload: unknown): DesktopSettings | null => {
if (typeof candidate.sttModel === 'string') {
result.sttModel = candidate.sttModel.trim();
}
if (typeof candidate.sttLocalModel === 'string') {
result.sttLocalModel = candidate.sttLocalModel.trim();
}
if (typeof candidate.sttLanguage === 'string') {
result.sttLanguage = candidate.sttLanguage.trim();
}
if (typeof candidate.sttSilenceThresholdDb === 'number' && Number.isFinite(candidate.sttSilenceThresholdDb)) {
result.sttSilenceThresholdDb = candidate.sttSilenceThresholdDb;
}
if (typeof candidate.sttSilenceHoldMs === 'number' && Number.isFinite(candidate.sttSilenceHoldMs)) {
result.sttSilenceHoldMs = candidate.sttSilenceHoldMs;
}
if (typeof candidate.sttTranscribeOnStop === 'boolean') {
result.sttTranscribeOnStop = candidate.sttTranscribeOnStop;
}
return result;
};
packages/ui/src/lib/settings/search.ts
+4 -10
View File
@@ -678,22 +678,16 @@ const SETTINGS_SEARCH_ITEMS: readonly SettingsSearchItem[] = [
keywords: ['keyboard', 'hotkeys', 'bindings'],
},
{
id: 'voice.voice-setup',
id: 'voice.playback',
page: 'voice',
titleKey: 'settings.voice.page.section.voiceSetup',
keywords: ['tts', 'voice mode', 'provider', 'speech rate', 'speech pitch', 'speech volume', 'language'],
titleKey: 'settings.voice.page.section.playbackAndSummary',
keywords: ['tts', 'read aloud', 'voice', 'provider', 'speech rate', 'speech pitch', 'speech volume', 'tts input mode', 'markdown'],
},
{
id: 'voice.speech-recognition',
page: 'voice',
titleKey: 'settings.voice.page.section.speechRecognition',
keywords: ['stt', 'transcribe', 'whisper', 'microphone', 'silence threshold'],
},
{
id: 'voice.playback',
page: 'voice',
titleKey: 'settings.voice.page.section.playbackAndSummary',
keywords: ['read aloud', 'tts input mode', 'summary', 'markdown'],
keywords: ['stt', 'dictation', 'voice input', 'transcribe', 'whisper', 'parakeet', 'microphone'],
},
{
id: 'tunnel.provider',
packages/ui/src/lib/shortcuts.ts
+7
View File
@@ -327,6 +327,13 @@ const SHORTCUT_ACTIONS: ReadonlyArray<ShortcutAction> = [
description: 'Toggle focus mode for the chat input',
customizable: true,
},
{
id: 'toggle_dictation',
defaultCombo: 'mod+alt+v',
label: 'Voice input',
description: 'Start dictation; press again to confirm and insert the transcript',
customizable: true,
},
{
id: 'abort_run',
defaultCombo: 'escape',
packages/ui/src/lib/voice/audioStreamService.ts
-397
View File
@@ -1,397 +0,0 @@
/**
* Audio Stream Service
*
* Captures microphone audio using MediaRecorder, detects utterance boundaries
* via an AnalyserNode-based silence detector (VAD), then POSTs each utterance
* as a raw audio blob to the OpenChamber server's /api/stt/transcribe endpoint.
*
* Mimics the BrowserVoiceService.startListening interface so useBrowserVoice
* can swap providers without changing its internal logic.
*
* @example
* ```typescript
* audioStreamService.configure({ baseURL: 'http://localhost:8001/v1', model: 'whisper-1' });
* audioStreamService.startListening('en', (text, isFinal) => {
* if (isFinal) console.log('transcript:', text);
* });
* audioStreamService.stopListening();
* ```
*/
import { runtimeFetch } from '@/lib/runtime-fetch';
type SpeechResultCallback = (text: string, isFinal: boolean) => void;
type ErrorCallback = (error: string) => void;
interface AudioStreamConfig {
/** Base URL of the OpenAI-compatible STT server (e.g. http://localhost:8001/v1) */
baseURL: string;
/** Whisper-compatible model name */
model: string;
/** Optional BCP-47 language hint (e.g. 'en'). Empty string = auto-detect. */
language?: string;
/**
* Silence threshold in dB below which audio is considered silence.
* Lower (more negative) = only very quiet audio counts as silence.
* Default: -45
*/
silenceThresholdDb?: number;
/**
* How long continuous silence must last (ms) before the utterance is finalised.
* Default: 1500
*/
silenceHoldMs?: number;
/** Optional API key for the STT server. */
apiKey?: string;
}
// How often (ms) the VAD samples the analyser
const VAD_POLL_MS = 80;
// Minimum audio duration (ms) to bother uploading (avoids blank clips)
const MIN_UTTERANCE_MS = 300;
class AudioStreamService {
private stream: MediaStream | null = null;
private mediaRecorder: MediaRecorder | null = null;
private audioContext: AudioContext | null = null;
private analyser: AnalyserNode | null = null;
private vadTimer: ReturnType<typeof setInterval> | null = null;
private chunks: Blob[] = [];
private recordingStartMs = 0;
private isActive = false;
private isSpeaking = false;
private silenceSince: number | null = null;
private onResult: SpeechResultCallback | null = null;
private onError: ErrorCallback | null = null;
private finishResolver: (() => void) | null = null;
private lang = 'en';
// Configurable parameters
private cfg: Required<AudioStreamConfig> = {
baseURL: '',
model: 'deepdml/faster-whisper-large-v3-turbo-ct2',
language: '',
silenceThresholdDb: -45,
silenceHoldMs: 1500,
apiKey: '',
};
/** Update service configuration. Can be called before or after startListening. */
configure(config: AudioStreamConfig): void {
this.cfg = {
silenceThresholdDb: -45,
silenceHoldMs: 1500,
language: '',
apiKey: '',
...config,
};
this.cfg.apiKey = config.apiKey ?? '';
}
/** Whether the browser supports the required APIs. */
isSupported(): boolean {
return (
typeof window !== 'undefined' &&
typeof navigator !== 'undefined' &&
typeof navigator.mediaDevices?.getUserMedia === 'function' &&
typeof window.MediaRecorder !== 'undefined' &&
typeof window.AudioContext !== 'undefined'
);
}
/**
* Start listening. Requests microphone access if not already held.
* Calls onResult(text, true) for each completed utterance.
*/
async startListening(
lang: string,
onResult: SpeechResultCallback,
onError?: ErrorCallback
): Promise<void> {
if (this.isActive) {
this.stopListening();
}
this.lang = lang;
this.onResult = onResult;
this.onError = onError ?? null;
this.isActive = true;
try {
this.stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
} catch (err) {
this.isActive = false;
const msg = err instanceof Error ? err.message : 'Microphone access denied';
onError?.(msg);
return;
}
this._setupAudioContext();
this._startRecorder();
this._startVAD();
}
/** Stop listening and clean up all resources. */
stopListening(): void {
this._stopVAD();
this._stopRecorder();
this._cleanupAfterStop(true);
}
async finishListening(): Promise<void> {
if (!this.isActive) return;
this._stopVAD();
this.isSpeaking = false;
this.silenceSince = null;
if (!this.mediaRecorder || this.mediaRecorder.state === 'inactive') {
this._cleanupAfterStop(true);
return;
}
await new Promise<void>((resolve) => {
this.finishResolver = resolve;
this._finaliseUtterance(false);
});
this._cleanupAfterStop(true);
}
/** Whether currently listening. */
getIsListening(): boolean {
return this.isActive;
}
// ── Private helpers ──────────────────────────────────────────────────────
private _setupAudioContext(): void {
if (!this.stream) return;
const AudioContextClass = window.AudioContext ?? (window as unknown as { webkitAudioContext: typeof AudioContext }).webkitAudioContext;
this.audioContext = new AudioContextClass();
const source = this.audioContext.createMediaStreamSource(this.stream);
this.analyser = this.audioContext.createAnalyser();
this.analyser.fftSize = 512;
source.connect(this.analyser);
}
private _teardownAudioContext(): void {
try {
this.audioContext?.close();
} catch {
// ignore
}
this.audioContext = null;
this.analyser = null;
}
private _startRecorder(): void {
if (!this.stream) return;
const mimeType = this._pickMimeType();
const options: MediaRecorderOptions = {};
if (mimeType && MediaRecorder.isTypeSupported(mimeType)) {
options.mimeType = mimeType;
}
this.mediaRecorder = new MediaRecorder(this.stream, options);
this.chunks = [];
this.recordingStartMs = Date.now();
this.mediaRecorder.ondataavailable = (e) => {
if (e.data && e.data.size > 0) {
this.chunks.push(e.data);
}
};
this.mediaRecorder.onstop = () => {
const blobs = this.chunks.splice(0);
const durationMs = Date.now() - this.recordingStartMs;
if (blobs.length === 0 || durationMs < MIN_UTTERANCE_MS) {
this.finishResolver?.();
this.finishResolver = null;
return;
}
const mType = blobs[0].type || mimeType || 'audio/webm';
const blob = new Blob(blobs, { type: mType });
void this._upload(blob, mType).finally(() => {
this.finishResolver?.();
this.finishResolver = null;
});
};
// Collect data every 250 ms so we don't lose the tail on stop()
this.mediaRecorder.start(250);
}
private _stopRecorder(): void {
if (this.mediaRecorder && this.mediaRecorder.state !== 'inactive') {
try {
this.mediaRecorder.stop();
} catch {
this.finishResolver?.();
this.finishResolver = null;
// ignore
}
}
this.mediaRecorder = null;
}
private _releaseStream(): void {
if (this.stream) {
this.stream.getTracks().forEach((t) => t.stop());
this.stream = null;
}
}
private _startVAD(): void {
this._stopVAD();
this.silenceSince = null;
this.isSpeaking = false;
this.vadTimer = setInterval(() => {
if (!this.isActive || !this.analyser) return;
const db = this._getRmsDb();
const isSilent = db < this.cfg.silenceThresholdDb;
if (!isSilent) {
// Audio detected
this.silenceSince = null;
if (!this.isSpeaking) {
this.isSpeaking = true;
// Restart recorder to capture from the start of speech
if (this.mediaRecorder?.state === 'recording') {
this.recordingStartMs = Date.now();
}
}
} else {
// Silence detected
if (this.isSpeaking) {
if (this.silenceSince === null) {
this.silenceSince = Date.now();
} else if (Date.now() - this.silenceSince >= this.cfg.silenceHoldMs) {
// End of utterance — stop recorder (triggers onstop → upload)
this.isSpeaking = false;
this.silenceSince = null;
this._finaliseUtterance(true);
}
}
}
}, VAD_POLL_MS);
}
private _stopVAD(): void {
if (this.vadTimer !== null) {
clearInterval(this.vadTimer);
this.vadTimer = null;
}
}
private _cleanupAfterStop(clearChunks: boolean): void {
const pendingResolver = this.finishResolver;
this.isActive = false;
this.finishResolver = null;
this.mediaRecorder = null;
this._teardownAudioContext();
this._releaseStream();
if (clearChunks) {
this.chunks = [];
}
this.isSpeaking = false;
this.silenceSince = null;
this.onResult = null;
this.onError = null;
pendingResolver?.();
}
/** Stop the current recorder to flush the utterance, optionally restarting for the next one. */
private _finaliseUtterance(restart: boolean): void {
if (!this.isActive) return;
if (this.mediaRecorder && this.mediaRecorder.state === 'recording') {
this.mediaRecorder.stop();
}
if (!restart) return;
// Restart recorder for the next utterance after a short delay
// (MediaRecorder.onstop fires asynchronously; we wait for it to complete)
setTimeout(() => {
if (this.isActive && this.stream) {
this._startRecorder();
}
}, 100);
}
/** Compute RMS of current analyser frame in dBFS. */
private _getRmsDb(): number {
if (!this.analyser) return -Infinity;
const buf = new Float32Array(this.analyser.fftSize);
this.analyser.getFloatTimeDomainData(buf);
let sumSq = 0;
for (const s of buf) sumSq += s * s;
const rms = Math.sqrt(sumSq / buf.length);
return rms === 0 ? -Infinity : 20 * Math.log10(rms);
}
/** POST utterance blob to server, call onResult with transcript. */
private async _upload(blob: Blob, mimeType: string): Promise<void> {
if (!this.onResult) return;
try {
const headers: Record<string, string> = {
'Content-Type': mimeType,
'X-Base-URL': this.cfg.baseURL,
'X-Model': this.cfg.model,
};
if (this.cfg.apiKey) {
headers['Authorization'] = `Bearer ${this.cfg.apiKey}`;
}
if (this.cfg.language) {
headers['X-Language'] = this.cfg.language;
} else if (this.lang && this.lang !== 'auto') {
// Use BCP-47 base language code (e.g. 'en' from 'en-US')
const baseLang = this.lang.split('-')[0];
headers['X-Language'] = baseLang;
}
const response = await runtimeFetch('/api/stt/transcribe', {
method: 'POST',
headers,
body: blob,
});
if (!response.ok) {
const errData = await response.json().catch(() => ({ error: 'Unknown error' }));
throw new Error(errData.error ?? `HTTP ${response.status}`);
}
const data = await response.json();
const transcript: string = (data.transcript ?? '').trim();
if (transcript) {
this.onResult(transcript, true);
}
} catch (err) {
if (!this.isActive) return; // Stopped — ignore
const msg = err instanceof Error ? err.message : 'Transcription upload failed';
console.error('[AudioStreamService] Upload error:', msg);
this.onError?.(msg);
}
}
/** Pick the best supported MIME type for MediaRecorder. */
private _pickMimeType(): string {
const candidates = [
'audio/webm;codecs=opus',
'audio/webm',
'audio/ogg;codecs=opus',
'audio/ogg',
'audio/mp4',
];
if (typeof MediaRecorder !== 'undefined' && MediaRecorder.isTypeSupported) {
return candidates.find((t) => MediaRecorder.isTypeSupported(t)) ?? '';
}
return '';
}
}
export const audioStreamService = new AudioStreamService();
packages/ui/src/lib/voice/contextFormatters.ts
-131
View File
@@ -1,131 +0,0 @@
/**
* Context formatters for voice-native output
* Formats session events (messages, permissions, ready events) into natural language
* for the ElevenLabs voice agent to speak aloud.
*
* @example
* ```typescript
* import { formatMessage, formatPermissionRequest } from '@/lib/voice';
*
* const voiceText = formatMessage({ role: 'assistant', content: 'Hello!' });
* // Returns: "Claude Code: Hello!"
* ```
*/
import { VOICE_CONFIG } from "./voiceConfig";
/** Message type for voice formatting */
export interface VoiceMessage {
role: string;
content: string;
}
/**
* Format a single message for voice output
* - Assistant messages: Code blocks replaced with "[code block]", prefixed with "Claude Code: "
* - User messages: Prefixed with "User: "
* - Other roles: Returns null (not spoken)
*
* @param message - The message to format
* @returns Formatted text for voice, or null if should not be spoken
*/
function formatMessage(message: VoiceMessage): string | null {
// Handle edge cases
if (!message || typeof message.content !== "string") {
return null;
}
const content = message.content.trim();
if (!content) {
return null;
}
if (message.role === "assistant") {
// Replace code blocks with description (don't read code aloud)
const textOnly = content.replace(/```[\s\S]*?```/g, "[code block]");
return `Claude Code: ${textOnly}`;
}
if (message.role === "user") {
return `User: ${content}`;
}
// Skip system, tool, and other roles for voice
return null;
}
/**
* Format multiple new messages for voice output
* - Maps messages through formatMessage
* - Filters out nulls (unspoken roles)
* - Joins with newlines
*
* @param sessionId - The session ID (for future use/debugging)
* @param messages - Array of messages to format
* @returns Formatted text for voice, or null if no speakable messages
*/
export function formatNewMessages(
sessionId: string,
messages: VoiceMessage[]
): string | null {
// Handle edge cases
if (!Array.isArray(messages) || messages.length === 0) {
return null;
}
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log(`[Voice] Formatting ${messages.length} messages for session ${sessionId}`);
}
// Format each message and filter out nulls
const formattedMessages = messages
.map(formatMessage)
.filter((msg): msg is string => msg !== null);
if (formattedMessages.length === 0) {
return null;
}
return formattedMessages.join("\n");
}
/**
* Format a permission request for voice announcement
* - Per CONTEXT.md: Only tool name, not arguments (LIMITED_TOOL_CALLS)
* - Prompts user to say "allow" or "deny"
*
* @param sessionId - The session ID
* @param requestId - The permission request ID
* @param toolName - Name of the tool requesting permission
* @param toolArgs - Tool arguments (not included in voice output per config)
* @returns Formatted permission request for voice
*/
export function formatPermissionRequest(
sessionId: string,
requestId: string,
toolName: string,
// eslint-disable-next-line @typescript-eslint/no-unused-vars
toolArgs: unknown
): string {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log(`[Voice] Formatting permission request ${requestId} for session ${sessionId}`);
}
// Per VOICE_CONFIG.LIMITED_TOOL_CALLS, we don't include toolArgs in voice output
return `Claude Code is requesting permission to use ${toolName}. Say "allow" or "deny".`;
}
/**
* Format a ready event for voice announcement
* - Indicates the AI has finished working and is ready for next instruction
*
* @param sessionId - The session ID
* @returns Formatted ready event for voice
*/
export function formatReadyEvent(sessionId: string): string {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log(`[Voice] Formatting ready event for session ${sessionId}`);
}
return "Claude Code finished working. Ready for next instruction.";
}
packages/ui/src/lib/voice/index.ts
-17
View File
@@ -1,17 +0,0 @@
/**
* Voice module barrel export
* Provides a clean import path for voice session hooks.
*
* @example
* ```typescript
* import { voiceHooks } from '@/lib/voice';
* ```
*/
// Voice session registry (from voiceSession.ts)
export {
isVoiceSessionStarted,
} from "./voiceSession";
// Voice hooks for session-to-voice event routing (from voiceHooks.ts)
export { voiceHooks } from "./voiceHooks";
packages/ui/src/lib/voice/summarize.ts
+10 -4
View File
@@ -6,15 +6,21 @@
export function sanitizeForTTS(text: string): string {
if (!text) return '';
return text
// Remove code blocks
// Remove fenced code blocks entirely (multi-line code is unreadable),
// but keep inline-code CONTENT and only strip the backticks: agents
// routinely inline meaningful words ("You are on `main`").
.replace(/```[\s\S]*?```/g, '')
.replace(/`[^`]*`/g, '')
.replace(/`([^`\n]*)`/g, '$1')
// Remove markdown formatting
.replace(/[*_~#]/g, '')
// Remove URLs
.replace(/https?:\/\/[^\s]+/g, '')
// Remove file paths
.replace(/\/[\w\-./]+/g, '')
// Remove absolute file paths (leading slash, one or more segments).
// Deliberately NOT matching interword slashes: "iOS/Android" and
// "origin/main" are speech, not paths.
.replace(/(^|\s)\/(?:[\w.-]+\/)*[\w.-]+/g, '$1')
// Read remaining interword slashes out loud ("iOS slash Android").
.replace(/([\w.])\/([\w.])/g, '$1 slash $2')
// Remove shell-like patterns
.replace(/^\s*[$#>]\s*/gm, '')
// Remove brackets and special chars
packages/ui/src/lib/voice/voiceConfig.ts
-32
View File
@@ -1,32 +0,0 @@
/**
* Static voice context configuration
* Controls voice behavior and feature flags for the ElevenLabs voice agent
*/
export const VOICE_CONFIG = {
/** Disable all tool call information from being sent to voice context */
DISABLE_TOOL_CALLS: false,
/** Send only tool names and descriptions, exclude arguments */
LIMITED_TOOL_CALLS: true,
/** Disable permission request forwarding */
DISABLE_PERMISSION_REQUESTS: false,
/** Disable session online/offline notifications */
DISABLE_SESSION_STATUS: true,
/** Disable message forwarding */
DISABLE_MESSAGES: false,
/** Disable session focus notifications */
DISABLE_SESSION_FOCUS: false,
/** Disable ready event notifications */
DISABLE_READY_EVENTS: false,
/** Maximum number of messages to include in session history */
MAX_HISTORY_MESSAGES: 50,
/** Enable debug logging for voice context updates */
ENABLE_DEBUG_LOGGING: true,
} as const;
packages/ui/src/lib/voice/voiceHooks.ts
-125
View File
@@ -1,125 +0,0 @@
/**
* Voice hooks for session-to-voice event routing
* Routes session events (messages, permissions, ready events) to the ElevenLabs
* voice agent via contextual updates.
*
* This module provides hooks that can be called when session events occur,
* using the voice session registry from voiceSession.ts.
*
* @example
* ```typescript
* import { voiceHooks } from '@/lib/voice';
*
* // Route session messages to voice
* voiceHooks.onMessages(sessionId, messages);
* ```
*/
import { VOICE_CONFIG } from "./voiceConfig";
import {
formatNewMessages,
formatPermissionRequest,
formatReadyEvent,
type VoiceMessage,
} from "./contextFormatters";
import { getVoiceSession, isVoiceSessionStarted } from "./voiceSession";
/**
* Report a contextual update to the voice session
* Internal helper that checks preconditions and handles errors
*
* @param update - The text update to send, or null/undefined to skip
*/
function reportContextualUpdate(update: string | null | undefined): void {
// Skip empty/null/undefined updates
if (!update || update.trim().length === 0) {
return;
}
// Skip if no voice session or not started
const voiceSession = getVoiceSession();
if (!voiceSession || !isVoiceSessionStarted()) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Skipping contextual update - no active session");
}
return;
}
try {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Sending contextual update:", update.substring(0, 100));
}
voiceSession.sendContextualUpdate(update);
} catch (error) {
// Log error but don't throw - voice updates shouldn't break the app
console.error("[Voice] Failed to send contextual update:", error);
}
}
/**
* Voice hooks - exported functions to route session events to voice
*
* These hooks should be called when corresponding session events occur.
* They respect VOICE_CONFIG feature flags to enable/disable specific
* event types.
*/
export const voiceHooks = {
/**
* Called when new messages arrive in the session
* Formats and sends messages to voice agent (if not disabled)
*
* @param sessionId - The session ID
* @param messages - Array of messages to format and send
*/
onMessages(sessionId: string, messages: VoiceMessage[]): void {
if (VOICE_CONFIG.DISABLE_MESSAGES) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Message forwarding disabled");
}
return;
}
reportContextualUpdate(formatNewMessages(sessionId, messages));
},
/**
* Called when a permission request is made
* Announces the permission request to the voice agent (if not disabled)
*
* @param sessionId - The session ID
* @param requestId - The permission request ID
* @param toolName - Name of the tool requesting permission
* @param toolArgs - Arguments for the tool (not sent to voice per LIMITED_TOOL_CALLS)
*/
onPermissionRequested(
sessionId: string,
requestId: string,
toolName: string,
toolArgs: unknown
): void {
if (VOICE_CONFIG.DISABLE_PERMISSION_REQUESTS) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Permission request forwarding disabled");
}
return;
}
reportContextualUpdate(
formatPermissionRequest(sessionId, requestId, toolName, toolArgs)
);
},
/**
* Called when the AI is ready for the next instruction
* Announces ready state to the voice agent (if not disabled)
*
* @param sessionId - The session ID
*/
onReady(sessionId: string): void {
if (VOICE_CONFIG.DISABLE_READY_EVENTS) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Ready event forwarding disabled");
}
return;
}
reportContextualUpdate(formatReadyEvent(sessionId));
},
};
packages/ui/src/lib/voice/voiceSession.ts
-28
View File
@@ -1,28 +0,0 @@
/**
* Voice session interface
* Used for type safety without importing ReturnType from SDK
*/
interface VoiceSession {
sendContextualUpdate: (text: string) => void;
}
/**
* Global storage for the active voice session.
* Used by voiceHooks to send contextual updates to the voice agent.
*/
const activeVoiceSession: VoiceSession | null = null;
/**
* Get the currently registered voice session.
* Used by voiceHooks to send contextual updates.
*/
export function getVoiceSession(): VoiceSession | null {
return activeVoiceSession;
}
/**
* Check if a voice session is currently active.
*/
export function isVoiceSessionStarted(): boolean {
return activeVoiceSession !== null;
}
packages/ui/src/lib/voice/wasmSttService.ts
-555
View File
@@ -1,555 +0,0 @@
/**
* WASM Speech-to-Text Service
*
* Local Whisper transcription via Transformers.js (ONNX Runtime Web).
* Captures microphone audio, detects utterance boundaries via silence-based
* VAD, then transcribes each utterance locally — no cloud API required.
*
* Works in Electron and all modern browsers that support Web Audio API.
* First use downloads a Whisper model (~40–166 MB, cached).
*/
export type WasmModelStatus =
| { state: 'unloaded' }
| { state: 'downloading'; progress: number }
| { state: 'loading' }
| { state: 'ready' }
| { state: 'error'; error: string };
export interface WasmModelInfo {
id: string;
name: string;
size: string;
languages: string;
description: string;
}
export const WASM_MODELS: WasmModelInfo[] = [
{
id: 'Xenova/whisper-tiny.en',
name: 'Whisper Tiny (EN)',
size: '~39 MB',
languages: 'English',
description: 'Fastest, lowest accuracy. Good for quick dictation.',
},
{
id: 'Xenova/whisper-base.en',
name: 'Whisper Base (EN)',
size: '~73 MB',
languages: 'English',
description: 'Balanced speed and accuracy. Default for English.',
},
{
id: 'Xenova/whisper-small.en',
name: 'Whisper Small (EN)',
size: '~166 MB',
languages: 'English',
description: 'Higher accuracy, slower. Best for noisy environments.',
},
];
type SpeechResultCallback = (text: string, isFinal: boolean) => void;
type ErrorCallback = (error: string) => void;
const VAD_POLL_MS = 80;
const MIN_UTTERANCE_MS = 300;
const WHISPER_SAMPLE_RATE = 16000;
interface WasmSttConfig {
silenceThresholdDb?: number;
silenceHoldMs?: number;
}
class WasmSttService {
private transcriber: unknown = null;
private worker: Worker | null = null;
private modelStatus: WasmModelStatus = { state: 'unloaded' };
private currentModelId: string | null = null;
private stream: MediaStream | null = null;
private mediaRecorder: MediaRecorder | null = null;
private audioContext: AudioContext | null = null;
private analyser: AnalyserNode | null = null;
private vadTimer: ReturnType<typeof setInterval> | null = null;
private chunks: Blob[] = [];
private recordingStartMs = 0;
private isActive = false;
private isSpeaking = false;
private silenceSince: number | null = null;
private onResult: SpeechResultCallback | null = null;
private onError: ErrorCallback | null = null;
private finishResolver: (() => void) | null = null;
private lang = 'en';
private cfg: Required<WasmSttConfig> = {
silenceThresholdDb: -45,
silenceHoldMs: 1500,
};
public onModelStatusChange: ((status: WasmModelStatus) => void) | null = null;
configure(config: WasmSttConfig): void {
this.cfg = { ...this.cfg, ...config };
}
isSupported(): boolean {
return (
typeof window !== 'undefined' &&
typeof navigator !== 'undefined' &&
typeof navigator.mediaDevices?.getUserMedia === 'function' &&
typeof window.MediaRecorder !== 'undefined' &&
typeof window.AudioContext !== 'undefined'
);
}
getModelStatus(): WasmModelStatus {
return this.modelStatus;
}
getCurrentModelId(): string | null {
return this.currentModelId;
}
private setModelStatus(status: WasmModelStatus): void {
this.modelStatus = status;
this.onModelStatusChange?.(status);
}
async loadModel(modelId: string): Promise<void> {
if (this.currentModelId === modelId && this.modelStatus.state === 'ready') {
return;
}
if (this.modelStatus.state === 'downloading' || this.modelStatus.state === 'loading') {
return;
}
this._terminateWorker();
this.transcriber = null;
this.setModelStatus({ state: 'downloading', progress: 0 });
this.currentModelId = modelId;
// Try Web Worker first — inference off main thread = no UI freeze.
try {
const WasmWorkerMod = await import('./wasmSttWorker?worker');
const WasmWorker = WasmWorkerMod.default as new () => Worker;
this.worker = new WasmWorker();
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('Worker init timed out')), 10000);
this.worker!.onmessage = (e: MessageEvent) => {
const data = e.data as { type: string; progress?: number; error?: string; text?: string };
if (data.type === 'progress') {
this.setModelStatus({ state: 'downloading', progress: data.progress ?? 0 });
} else if (data.type === 'loaded') {
clearTimeout(timer);
resolve();
} else if (data.type === 'error') {
clearTimeout(timer);
reject(new Error(data.error ?? 'Worker load failed'));
}
};
this.worker!.onerror = (err) => {
clearTimeout(timer);
reject(new Error(err.message || 'Worker error'));
};
this.worker!.postMessage({ type: 'load', modelId });
});
this.setModelStatus({ state: 'ready' });
return;
} catch (err) {
console.warn('[WasmStt] Worker failed, using main-thread:', err instanceof Error ? err.message : err);
this._terminateWorker();
}
// Fallback: main-thread pipeline (causes brief UI freeze during inference).
try {
const { pipeline, env } = await import('@xenova/transformers');
env.backends.onnx.wasm.numThreads = 1;
env.allowLocalModels = false;
const fileDoneBytes = new Map<string, number>();
let totalDone = 0;
let totalEstimate = 0;
this.transcriber = await pipeline('automatic-speech-recognition', modelId, {
progress_callback: (info: { status?: string; file?: string; loaded?: number; total?: number }) => {
if (info.status === 'progress' && info.file) {
const prevDone = fileDoneBytes.get(info.file) ?? 0;
const currentDone = info.loaded ?? 0;
const delta = Math.max(0, currentDone - prevDone);
fileDoneBytes.set(info.file, currentDone);
totalDone += delta;
if (info.total && info.total > totalEstimate) totalEstimate = info.total;
const effectiveTotal = Math.max(totalEstimate, totalDone);
const pct = effectiveTotal > 0 ? Math.min(100, Math.round((totalDone / effectiveTotal) * 100)) : 0;
this.setModelStatus({ state: 'downloading', progress: pct });
}
},
});
this.setModelStatus({ state: 'ready' });
} catch (err) {
const msg = err instanceof Error ? err.message : 'Unknown error loading model';
this.setModelStatus({ state: 'error', error: msg });
this.transcriber = null;
this.currentModelId = null;
throw err;
}
}
private _terminateWorker(): void {
if (this.worker) {
this.worker.terminate();
this.worker = null;
}
}
async unloadModel(): Promise<void> {
this._terminateWorker();
this.transcriber = null;
this.currentModelId = null;
this.setModelStatus({ state: 'unloaded' });
}
async startListening(
lang: string,
onResult: SpeechResultCallback,
onError?: ErrorCallback,
): Promise<void> {
if (this.isActive) {
this.stopListening();
}
if (!this.transcriber && !this.worker) {
onError?.('Whisper model not loaded. Select a model in Voice Settings first.');
return;
}
this.lang = lang;
this.onResult = onResult;
this.onError = onError ?? null;
this.isActive = true;
try {
this.stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
} catch (err) {
this.isActive = false;
const msg = err instanceof Error ? err.message : 'Microphone access denied';
onError?.(msg);
return;
}
this._setupAudioContext();
this._startRecorder();
this._startVAD();
}
stopListening(): void {
this._stopVAD();
if (this.mediaRecorder && this.mediaRecorder.state === 'recording') {
try { this.mediaRecorder.stop(); } catch { /* ignore */ }
}
this._cleanupAfterStop(true);
}
async finishListening(): Promise<void> {
if (!this.isActive) return;
this._stopVAD();
this.isSpeaking = false;
this.silenceSince = null;
if (!this.mediaRecorder || this.mediaRecorder.state === 'inactive') {
this._cleanupAfterStop(true);
return;
}
await new Promise<void>((resolve) => {
this.finishResolver = resolve;
this._finaliseUtterance(false);
});
this._cleanupAfterStop(true);
}
getIsListening(): boolean {
return this.isActive;
}
// ── Audio capture ────────────────────────────────────────────────────
private _setupAudioContext(): void {
if (!this.stream) return;
const AudioContextClass = window.AudioContext ?? (window as unknown as { webkitAudioContext: typeof AudioContext }).webkitAudioContext;
this.audioContext = new AudioContextClass();
const source = this.audioContext.createMediaStreamSource(this.stream);
this.analyser = this.audioContext.createAnalyser();
this.analyser.fftSize = 512;
source.connect(this.analyser);
}
private _teardownAudioContext(): void {
try { this.audioContext?.close(); } catch { /* ignore */ }
this.audioContext = null;
this.analyser = null;
}
private _startRecorder(): void {
if (!this.stream) return;
const mimeType = this._pickMimeType();
const options: MediaRecorderOptions = {};
if (mimeType && MediaRecorder.isTypeSupported(mimeType)) {
options.mimeType = mimeType;
}
this.mediaRecorder = new MediaRecorder(this.stream, options);
this.chunks = [];
this.recordingStartMs = Date.now();
this.mediaRecorder.ondataavailable = (e) => {
if (e.data && e.data.size > 0) {
this.chunks.push(e.data);
}
};
this.mediaRecorder.onstop = () => {
const blobs = this.chunks.splice(0);
const durationMs = Date.now() - this.recordingStartMs;
if (blobs.length === 0 || durationMs < MIN_UTTERANCE_MS) {
this.finishResolver?.();
this.finishResolver = null;
return;
}
const mType = blobs[0].type || mimeType || 'audio/webm';
const blob = new Blob(blobs, { type: mType });
void this._transcribe(blob).finally(() => {
this.finishResolver?.();
this.finishResolver = null;
});
};
this.mediaRecorder.start(250);
}
private _releaseStream(): void {
if (this.stream) {
this.stream.getTracks().forEach((t) => t.stop());
this.stream = null;
}
}
// ── VAD ──────────────────────────────────────────────────────────────
private _startVAD(): void {
this._stopVAD();
this.silenceSince = null;
this.isSpeaking = false;
this.vadTimer = setInterval(() => {
if (!this.isActive || !this.analyser) return;
const db = this._getRmsDb();
const isSilent = db < this.cfg.silenceThresholdDb;
if (!isSilent) {
this.silenceSince = null;
if (!this.isSpeaking) {
this.isSpeaking = true;
if (this.mediaRecorder?.state === 'recording') {
this.recordingStartMs = Date.now();
}
}
} else {
if (this.isSpeaking) {
if (this.silenceSince === null) {
this.silenceSince = Date.now();
} else if (Date.now() - this.silenceSince >= this.cfg.silenceHoldMs) {
this.isSpeaking = false;
this.silenceSince = null;
this._finaliseUtterance(true);
}
}
}
}, VAD_POLL_MS);
}
private _stopVAD(): void {
if (this.vadTimer !== null) {
clearInterval(this.vadTimer);
this.vadTimer = null;
}
}
private _cleanupAfterStop(clearChunks: boolean): void {
const pendingResolver = this.finishResolver;
this.isActive = false;
this.finishResolver = null;
this.mediaRecorder = null;
this._teardownAudioContext();
this._releaseStream();
if (clearChunks) this.chunks = [];
this.isSpeaking = false;
this.silenceSince = null;
this.onResult = null;
this.onError = null;
pendingResolver?.();
}
private _finaliseUtterance(restart: boolean): void {
if (!this.isActive) return;
if (this.mediaRecorder && this.mediaRecorder.state === 'recording') {
this.mediaRecorder.stop();
}
if (!restart) return;
setTimeout(() => {
if (this.isActive && this.stream) {
this._startRecorder();
}
}, 100);
}
private _getRmsDb(): number {
if (!this.analyser) return -Infinity;
const buf = new Float32Array(this.analyser.fftSize);
this.analyser.getFloatTimeDomainData(buf);
let sumSq = 0;
for (const s of buf) sumSq += s * s;
const rms = Math.sqrt(sumSq / buf.length);
return rms === 0 ? -Infinity : 20 * Math.log10(rms);
}
// ── Transcription ────────────────────────────────────────────────────
private async _transcribe(blob: Blob): Promise<void> {
if (!this.onResult) return;
if (!this.transcriber && !this.worker) {
this.onError?.('Model not loaded');
return;
}
try {
const audioData = await this._decodeToFloat32(blob);
if (!audioData || audioData.length === 0) {
this.onError?.(`Failed to decode audio (${blob.size} bytes)`);
return;
}
const langHint = this._resolveLanguageHint();
// Prefer worker (non-blocking); fall back to main-thread pipeline.
const transcript = this.worker
? await this._transcribeViaWorker(audioData, langHint)
: await this._transcribeMainThread(audioData, langHint);
if (transcript) {
this.onResult(transcript, true);
}
} catch (err) {
if (!this.isActive) return;
const msg = err instanceof Error ? err.message : 'Local transcription failed';
this.onError?.(msg);
}
}
private _transcribeViaWorker(audioData: Float32Array, langHint: string | undefined): Promise<string> {
return new Promise((resolve, reject) => {
if (!this.worker) return reject(new Error('Worker gone'));
const onMessage = (e: MessageEvent) => {
const data = e.data as { type: string; error?: string; transcript?: string; text?: string };
if (data.type === 'result') {
this.worker!.removeEventListener('message', onMessage);
resolve(data.transcript ?? '');
} else if (data.type === 'log') {
console.log('[WasmStt Worker]', data.text);
} else if (data.type === 'error') {
this.worker!.removeEventListener('message', onMessage);
reject(new Error(data.error ?? 'Transcription failed'));
}
};
this.worker.addEventListener('message', onMessage);
this.worker.postMessage(
{ type: 'transcribe', audio: audioData.buffer, language: langHint },
[audioData.buffer],
);
setTimeout(() => {
this.worker?.removeEventListener('message', onMessage);
reject(new Error('Transcription timed out'));
}, 30000);
});
}
private async _transcribeMainThread(audioData: Float32Array, langHint: string | undefined): Promise<string> {
const pipelineFn = this.transcriber as (
input: Float32Array,
options?: Record<string, unknown>,
) => Promise<{ text: string }>;
const result = await pipelineFn(audioData, {
task: 'transcribe',
...(langHint ? { language: langHint } : {}),
});
return (result?.text ?? '').trim();
}
private async _decodeToFloat32(blob: Blob): Promise<Float32Array | null> {
if (!this.audioContext) return null;
const arrayBuffer = await blob.arrayBuffer();
let audioBuffer: AudioBuffer;
try {
audioBuffer = await this.audioContext.decodeAudioData(arrayBuffer);
} catch {
return null;
}
const origRate = audioBuffer.sampleRate;
const origData = audioBuffer.getChannelData(0);
const targetRate = WHISPER_SAMPLE_RATE;
if (origRate === targetRate) {
return new Float32Array(origData);
}
const ratio = origRate / targetRate;
const newLength = Math.ceil(origData.length / ratio);
const result = new Float32Array(newLength);
for (let i = 0; i < newLength; i++) {
const origIdx = i * ratio;
const idx0 = Math.floor(origIdx);
const idx1 = Math.min(idx0 + 1, origData.length - 1);
const frac = origIdx - idx0;
result[i] = origData[idx0] * (1 - frac) + origData[idx1] * frac;
}
return result;
}
private _resolveLanguageHint(): string | undefined {
if (this.lang && this.lang !== 'auto') {
return this.lang.split('-')[0];
}
return undefined;
}
private _pickMimeType(): string {
const candidates = [
'audio/webm;codecs=opus',
'audio/webm',
'audio/ogg;codecs=opus',
'audio/ogg',
'audio/mp4',
];
if (typeof MediaRecorder !== 'undefined' && MediaRecorder.isTypeSupported) {
return candidates.find((t) => MediaRecorder.isTypeSupported(t)) ?? '';
}
return '';
}
}
export const wasmSttService = new WasmSttService();
packages/ui/src/lib/voice/wasmSttWorker.ts
-92
View File
@@ -1,92 +0,0 @@
/**
* Web Worker for off-main-thread Whisper transcription.
*
* Receives `{ type: 'load', modelId }` to load a model, then
* `{ type: 'transcribe', audio: Float32Array (transferred buffer), language? }`
* to run inference. Posts progress, results, and errors back.
*/
import { pipeline, env } from '@xenova/transformers';
let transcriber: unknown = null;
self.onmessage = async (e: MessageEvent) => {
const { type } = e.data as { type: string };
if (type === 'load') {
const { modelId } = e.data as { modelId: string };
try {
env.backends.onnx.wasm.numThreads = 1;
const fileDoneBytes = new Map<string, number>();
let totalDone = 0;
let totalEstimate = 0;
transcriber = await pipeline('automatic-speech-recognition', modelId, {
progress_callback: (info: { status?: string; file?: string; loaded?: number; total?: number }) => {
if (info.status === 'progress' && info.file) {
const prevDone = fileDoneBytes.get(info.file) ?? 0;
const currentDone = info.loaded ?? 0;
const delta = Math.max(0, currentDone - prevDone);
fileDoneBytes.set(info.file, currentDone);
totalDone += delta;
if (info.total && info.total > totalEstimate) {
totalEstimate = info.total;
}
const effectiveTotal = Math.max(totalEstimate, totalDone);
const pct = effectiveTotal > 0 ? Math.min(100, Math.round((totalDone / effectiveTotal) * 100)) : 0;
self.postMessage({ type: 'progress', progress: pct });
}
},
});
self.postMessage({ type: 'loaded' });
} catch (err) {
self.postMessage({
type: 'error',
error: err instanceof Error ? err.message : 'Failed to load model',
});
}
} else if (type === 'transcribe') {
if (!transcriber) {
self.postMessage({ type: 'error', error: 'Model not loaded', seq: (e.data as { seq?: number }).seq });
return;
}
const { audio, language, seq } = e.data as { audio: ArrayBuffer; language?: string; seq?: number };
try {
const samples = new Float32Array(audio);
if (samples.length === 0) {
self.postMessage({ type: 'error', error: 'Empty audio received', seq });
return;
}
self.postMessage({ type: 'log', text: `Transcribing ${samples.length} samples (${(samples.length / 16000).toFixed(1)}s)` });
const pipelineFn = transcriber as (
input: Float32Array,
options?: Record<string, unknown>,
) => Promise<{ text: string }>;
const result = await pipelineFn(samples, {
task: 'transcribe',
...(language ? { language } : {}),
});
self.postMessage({
type: 'result',
transcript: (result?.text ?? '').trim(),
seq,
});
} catch (err) {
self.postMessage({
type: 'error',
error: err instanceof Error ? err.message : 'Transcription failed',
seq,
});
}
}
};