Files
openchamber/packages/ui/src/lib/voice/voiceHooks.ts
T
gsxdsmandBohdan Triapitsyn 1ed5316ac7 feat(voice): add voice input/output support with multiple providers (#281)
* feat(voice): add voice input/output support with multiple providers

- Add BrowserVoiceButton component for Web Speech API voice input
- Add VoiceProvider context for managing voice state across the app
- Add TTS (Text-to-Speech) support with browser, macOS Say, and OpenAI providers
- Add message TTS buttons to read assistant messages aloud
- Add VoiceSettings page in OpenChamber settings
- Add server endpoints for TTS and summarization services
- Include slider component for voice rate/pitch/volume controls
- Add hidden session support for background voice operations
- Add Caddyfile for HTTPS support (required for microphone access)

* fix: Build errors fixed and removed outdated ElevenLabs test code.

* refactor(voice): use zen API with gpt-5-nano for TTS summarization

Replace the hidden session + OpenCode SDK approach with direct calls
to the opencode.ai zen API (same pattern used for commit message and
PR description generation).

- Rewrite summarization-service.js to call zen/v1/responses with gpt-5-nano
- Remove hidden session logic (hiddenSession.ts, sessionStore filtering)
- Remove summarizeModel setting and model selector from VoiceSettings
- Simplify client-side summarize.ts to no longer pass model params
- Clean up callers in useMessageTTS and useBrowserVoice

* fix(voice): remove false 'voice not supported' warning in settings

Mobile Safari does support voice but the isSupported check was
incorrectly flagging it. Remove the warning banner entirely.

* feat(voice): add configurable summary length limit for TTS output

Add a slider (50-2000 chars) in voice settings to control max summary
length. The limit is passed through the summarize endpoint and speak
endpoint to the zen API prompt, with token budget scaled accordingly.

* fix(voice): add diagnostic logging and sanitize TTS fallback

Add console logging throughout the summarization flow (client + server)
to trace why text may not be summarized. Fix silent error swallowing in
/api/tts/speak. Always apply sanitizeForTTS even when summarization is
disabled so raw markdown/code is never spoken verbatim.

* fix(voice): fix token budget starving model of output tokens

max_output_tokens includes both reasoning and output tokens. With
effort:'low', reasoning alone consumes ~128 tokens, so a budget of
100 left zero tokens for the actual summary text. Use a fixed 1000
token budget (matching commit message generation) and control output
length via the prompt's character limit instruction instead.

* chore(voice): remove diagnostic logging from summarization flow

* fix(voice): don't request mic permission on mobile page load

Remove the useEffect that pre-requested microphone permission when the
BrowserVoiceButton component mounted on mobile. This caused an unwanted
permission prompt immediately on page load before the user tapped the
mic icon. Permission is now only requested on explicit user interaction.

* fix(voice): remove unused BrowserVoiceButton binding

* fix(voice): desktop mic flow + non-continuous draft mode

* fix(voice): stabilize continuous loop and polish controls

* feat(settings): mark voice section experimental

---------

Co-authored-by: Bohdan Triapitsyn <artmore@protonmail.com>
2026-02-09 23:55:10 +02:00

134 lines
4.2 KiB
TypeScript

/**
* Voice hooks for session-to-voice event routing
* Routes session events (messages, permissions, ready events) to the ElevenLabs
* voice agent via contextual updates.
*
* This module provides hooks that can be called when session events occur,
* using the voice session registry from voiceSession.ts.
*
* @example
* ```typescript
* import { voiceHooks } from '@/lib/voice';
*
* // Route session messages to voice
* voiceHooks.onMessages(sessionId, messages);
* ```
*/
import { VOICE_CONFIG } from "./voiceConfig";
import {
formatNewMessages,
formatPermissionRequest,
formatReadyEvent,
type VoiceMessage,
} from "./contextFormatters";
import { getVoiceSession, isVoiceSessionStarted } from "./voiceSession";
// Re-export registry functions from voiceSession.ts for convenience
export {
registerVoiceSession,
unregisterVoiceSession,
getVoiceSession,
isVoiceSessionStarted,
} from "./voiceSession";
/**
* Report a contextual update to the voice session
* Internal helper that checks preconditions and handles errors
*
* @param update - The text update to send, or null/undefined to skip
*/
function reportContextualUpdate(update: string | null | undefined): void {
// Skip empty/null/undefined updates
if (!update || update.trim().length === 0) {
return;
}
// Skip if no voice session or not started
const voiceSession = getVoiceSession();
if (!voiceSession || !isVoiceSessionStarted()) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Skipping contextual update - no active session");
}
return;
}
try {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Sending contextual update:", update.substring(0, 100));
}
voiceSession.sendContextualUpdate(update);
} catch (error) {
// Log error but don't throw - voice updates shouldn't break the app
console.error("[Voice] Failed to send contextual update:", error);
}
}
/**
* Voice hooks - exported functions to route session events to voice
*
* These hooks should be called when corresponding session events occur.
* They respect VOICE_CONFIG feature flags to enable/disable specific
* event types.
*/
export const voiceHooks = {
/**
* Called when new messages arrive in the session
* Formats and sends messages to voice agent (if not disabled)
*
* @param sessionId - The session ID
* @param messages - Array of messages to format and send
*/
onMessages(sessionId: string, messages: VoiceMessage[]): void {
if (VOICE_CONFIG.DISABLE_MESSAGES) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Message forwarding disabled");
}
return;
}
reportContextualUpdate(formatNewMessages(sessionId, messages));
},
/**
* Called when a permission request is made
* Announces the permission request to the voice agent (if not disabled)
*
* @param sessionId - The session ID
* @param requestId - The permission request ID
* @param toolName - Name of the tool requesting permission
* @param toolArgs - Arguments for the tool (not sent to voice per LIMITED_TOOL_CALLS)
*/
onPermissionRequested(
sessionId: string,
requestId: string,
toolName: string,
toolArgs: unknown
): void {
if (VOICE_CONFIG.DISABLE_PERMISSION_REQUESTS) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Permission request forwarding disabled");
}
return;
}
reportContextualUpdate(
formatPermissionRequest(sessionId, requestId, toolName, toolArgs)
);
},
/**
* Called when the AI is ready for the next instruction
* Announces ready state to the voice agent (if not disabled)
*
* @param sessionId - The session ID
*/
onReady(sessionId: string): void {
if (VOICE_CONFIG.DISABLE_READY_EVENTS) {
if (VOICE_CONFIG.ENABLE_DEBUG_LOGGING) {
console.log("[Voice] Ready event forwarding disabled");
}
return;
}
reportContextualUpdate(formatReadyEvent(sessionId));
},
};