Bohdan Triapitsyn
28f0736d69
Adds a server-side "small model" capability: direct, cheap LLM calls that reuse the user's existing OpenCode provider logins — the mechanism OpenCode uses internally for titles and summaries but does not expose through the SDK or plugins. Zero new dependencies; plain fetch with per-provider wire formats, credentials never leave the server. Core (packages/web/server/lib/small-model): - Resolution mirrors OpenCode's session scoping: explicit settings override → small_model from the OpenCode config → family scan within the session's provider → the session's own model. The global provider scan only serves callers without a session context, and background callers forbid it entirely (restrictToPreferredProvider), so conversation content never reaches a provider the user didn't pick — explicit choices excepted. - Per-provider auth replicating OpenCode's plugin loaders: GitHub Copilot (device token as bearer, no exchange), ChatGPT plan via the codex Responses API (single-flight OAuth refresh written back to auth.json), Anthropic messages, Google generateContent, generic OpenAI-compatible. - OpenCode's free models (opencode/big-pickle, *-free) are never called directly; unauthenticated providers are skipped by design. - Prompt clamping to the model's catalog context limit; thinking disabled where a wire switch exists (Z.AI/GLM, MiniMax-M3, Gemini Flash); robust content parsing with a clear error when a thinking model spends its whol budget on reasoning. - Settings → Sessions gains a Small Model group: use-default checkbox plus an override picker limited to authenticated providers, persisted with web/desktop/VS Code sanitization parity. Consumers: - Session assist: a server-side watcher on the global SSE hub generates a short recap and one suggested follow-up after a session idles quietly fo a minute, stored on session metadata (openchamber.assist). Freshness is keyed to the last assistant message id, so new activity invalidates the payload everywhere with no extra writes. The chat shows the recap under the last message after five quiet minutes and the suggestion as a dismissible chip above the composer (tap fills the input, never sends). Gated by a new Chat setting (default on) that is a hard generation switch. Language is anchored to the conversation itself, with a script-mismatch guard against model/backend language hallucination. - TTS: a third input mode, summarized — long replies are condensed to spoken prose before playback on any TTS engine. - Git: commit-message and PR generation moved off the active chat session onto the small model fed with real diffs and the commit list (bodies included), with a session-transport fallback for free-model-only setups. - Notes: Add to notes distills long selections into 1-3 dense sentences preserving exact identifiers, with verbatim fallback on failure. Fixes along the way: - The global event watcher now starts unconditionally; it was gated behind the desktop-notify env, leaving the server-side event hub dead in packaged apps. - OpenCode re-emits message.updated for old user messages after idle; the watcher no longer mistakes those for new activity. - Session metadata merges from a fresh read right before the PATCH, so writes made during the generation window (suggestion dismissals, review links) are preserved; the assist runtime stops during graceful shutdown.
79 lines
4.3 KiB
Markdown
79 lines
4.3 KiB
Markdown
# Small Model
|
|
|
|
Server-side direct LLM calls that reuse the user's existing OpenCode provider
|
|
logins (`~/.local/share/opencode/auth.json`). OpenCode uses a "small model"
|
|
internally (titles, summaries) but does not expose it through the SDK or
|
|
plugins — this module replicates that mechanism as an OpenChamber runtime API.
|
|
|
|
## Security boundary
|
|
|
|
Credentials never leave the server process. The client sends only a prompt;
|
|
auth resolution, OAuth refresh, and provider dispatch all happen server-side.
|
|
Routes live under `/api/*` and are gated by the ui-auth middleware like every
|
|
other runtime API.
|
|
|
|
## Files
|
|
|
|
- `index.js` — orchestration: `generateSmallModelText()` / `describeSmallModel()`.
|
|
- `resolve.js` — model selection, mirroring OpenCode's `getSmallModel` chain:
|
|
0. OpenChamber's own settings override (Settings → Sessions → Small Model):
|
|
when `smallModelUseDefault` is `false`, `smallModelOverride`
|
|
(`provider/model`) outranks everything below. Sanitized in
|
|
`settings-helpers.js` (server), `persistence.ts` (client), and
|
|
`bridge-settings-runtime.ts` (VS Code).
|
|
1. `small_model` from the merged OpenCode config layers (`provider/model`).
|
|
2. Family-priority scan (`gemini-flash` → `gpt-nano` → `claude-haiku`)
|
|
**within the session's provider first** (`preferredProviderID`, like
|
|
OpenCode resolves within the current provider), then over the other
|
|
providers with a usable auth entry, newest `release_date` first.
|
|
3. GitHub Copilot hidden utility models (`gpt-*-nano/mini`) — these never
|
|
appear in the catalog, so they participate as the `gpt-nano` family entry
|
|
and as a final utility fallback.
|
|
4. Last resort: the session's own model (`preferredModelID`) when no small
|
|
model resolves anywhere — costlier, but always valid.
|
|
- Input clamp: the prompt is truncated to the resolved model's catalog
|
|
`limit.context` (minus an output reserve, ~4 chars/token estimate;
|
|
conservative default when the model is not in the catalog). Truncation is
|
|
reported as `inputTruncated: true` in the response.
|
|
- `call.js` — wire formats and per-provider auth, replicating OpenCode's
|
|
plugin auth loaders:
|
|
- **GitHub Copilot**: OpenAI-compatible `/chat/completions` on
|
|
`https://api.githubcopilot.com` (or `copilot-api.<enterprise>`) with the
|
|
stored device-OAuth token as the bearer — no token exchange, no expiry.
|
|
- **OpenAI OAuth (ChatGPT plan)**: streaming Responses API on
|
|
`https://chatgpt.com/backend-api/codex/responses` with
|
|
`ChatGPT-Account-Id`; expired tokens are refreshed against
|
|
`auth.openai.com` (single-flight) and written back to `auth.json`.
|
|
- **Anthropic** (`type: api`): `/v1/messages` with `x-api-key`.
|
|
- **Google** (`type: api`): `generateContent` with `x-goog-api-key`.
|
|
- Everything else: OpenAI-compatible `/chat/completions` against the
|
|
provider's models.dev base URL with `Authorization: Bearer <key>`.
|
|
- `catalog.js` — models.dev catalog via the shared in-process cache
|
|
(`../opencode/models-metadata.js`, also serving
|
|
`/api/openchamber/models-metadata`).
|
|
- `routes.js` — `GET /api/small-model` (resolution preview) and
|
|
`POST /api/small-model/generate` (`{ prompt, system?, maxOutputTokens?,
|
|
model?, directory? }` → `{ text, providerID, modelID, source }`).
|
|
|
|
## Registration
|
|
|
|
Mounted lazily from `feature-routes-runtime.js` (same pattern as quota): the
|
|
module is imported on first request, not at server startup.
|
|
|
|
## Known limitations
|
|
|
|
- OpenCode's free models (`opencode/big-pickle`, `*-free`) work without a
|
|
token only through OpenCode's own server — direct calls are rejected, and
|
|
piggybacking on their subsidized infra is out of bounds by design. Every
|
|
resolution step therefore requires a usable auth entry for the provider:
|
|
a session on an unauthenticated `opencode` provider falls through to the
|
|
global scan (or a clean 404 on a vanilla setup with no logins).
|
|
|
|
- Anthropic OAuth (Claude Pro/Max) entries are not supported — OpenCode itself
|
|
keeps those outside `auth.json` in this generation; only `type: api` keys
|
|
work for Anthropic.
|
|
- Amazon Bedrock, GitLab, Azure and other credential-chain providers are out
|
|
of scope; they need more than a key/token (regions, resource names).
|
|
- Responses from the codex backend are collected from the SSE stream; the
|
|
endpoint itself is non-streaming by design (small utility calls).
|