Serhii Dziupin
8a85073261
OpenChamber's Settings → Chat → Small Model override only fed OpenChamber's own /api/small-model/generate utility service; it never reached the managed OpenCode server, whose internal title/summary generation reads small_model from its config. With the override injected into OPENCODE_CONFIG_CONTENT at managed-process launch, session title generation uses the user's explicit model instead of falling back (or failing to resolve) — fixing sessions that stayed untitled even with a Small Model configured. Only an explicit override (smallModelUseDefault === false with a non-empty smallModelOverride) is injected; "use default" leaves the config untouched so OpenCode's own resolution chain stays authoritative. Malformed user config is left unmodified. External OpenCode servers are unaffected (they are not launched with this env). Fixes #2497
150 lines
9.2 KiB
Markdown
150 lines
9.2 KiB
Markdown
# Small Model
|
|
|
|
Server-side direct LLM calls that reuse the user's existing OpenCode provider
|
|
logins (`~/.local/share/opencode/auth.json`). OpenCode uses a "small model"
|
|
internally (titles, summaries) but does not expose it through the SDK or
|
|
plugins — this module replicates that mechanism as an OpenChamber runtime API.
|
|
|
|
## Security boundary
|
|
|
|
Credentials never leave the server process. The client sends only a prompt;
|
|
auth resolution, OAuth refresh, and provider dispatch all happen server-side.
|
|
Routes live under `/api/*` and are gated by the ui-auth middleware like every
|
|
other runtime API.
|
|
|
|
## Files
|
|
|
|
- `index.js` — orchestration: `generateSmallModelText()` / `describeSmallModel()`.
|
|
- `resolve.js` — model selection, mirroring OpenCode's `getSmallModel` chain:
|
|
0. OpenChamber's own settings override (Settings → Sessions → Small Model):
|
|
when `smallModelUseDefault` is `false`, `smallModelOverride`
|
|
(`provider/model`) outranks everything below. Sanitized in
|
|
`settings-helpers.js` (server), `persistence.ts` (client), and
|
|
`bridge-settings-runtime.ts` (VS Code).
|
|
1. `small_model` from the merged OpenCode config layers (`provider/model`).
|
|
2. Family-priority scan (`gemini-flash` → `gpt-nano` → `claude-haiku`)
|
|
**within the session's provider first** (`preferredProviderID`, like
|
|
OpenCode resolves within the current provider), then over the other
|
|
providers with a usable auth entry, newest `release_date` first.
|
|
3. GitHub Copilot hidden utility models (`gpt-*-nano/mini`) — these never
|
|
appear in the catalog, so they participate as the `gpt-nano` family entry
|
|
and as a final utility fallback.
|
|
4. Last resort: the session's own model (`preferredModelID`) when no small
|
|
model resolves anywhere — costlier, but always valid.
|
|
- Input clamp: the prompt is measured against the resolved model's catalog
|
|
`limit.context` (minus an output reserve, ~4 chars/token estimate;
|
|
conservative default when the model is not in the catalog). `onOverflow`
|
|
decides what an oversized prompt means:
|
|
- `truncate` (default) clips the tail and reports `inputTruncated: true`.
|
|
Correct for callers that degrade gracefully (summaries, commit messages).
|
|
- `error` throws a `413` with `code: 'context-too-small'` plus
|
|
`requiredChars`/`availableChars`. Correct for callers whose output would be
|
|
quietly wrong on a clipped input, so they can ask the user for a roomier
|
|
model instead of returning confident nonsense.
|
|
- Structured output: pass `responseSchema` (a JSON Schema) to get
|
|
schema-shaped JSON back as `text`. Wire support differs per format —
|
|
`response_format: {type: 'json_schema'}` for OpenAI-compatible chat,
|
|
`text.format` for the Responses API, a forced single tool call for the
|
|
Anthropic messages API, and `generationConfig.responseSchema` for Google
|
|
(whose OpenAPI-flavored dialect drops unknown JSON Schema keywords). The
|
|
ChatGPT-plan codex backend has no equivalent and rejects a schema request
|
|
with `code: 'structured-output-unsupported'` rather than silently returning
|
|
prose.
|
|
- Output budget: `maxOutputTokens` is capped at the catalog's `limit.output` for
|
|
the model, and the **same number** is reserved from the input allowance. The
|
|
two must not drift — a caller that asks for a large answer while the reserve
|
|
stays at the default overruns the context, and the failure looks like a
|
|
truncation bug rather than a budgeting one. `describeSmallModel` takes
|
|
`outputReserveTokens` so readiness checks agree with what generation will do.
|
|
It may be a **function** of `{ contextTokens, outputTokenLimit }` for callers
|
|
that want as much answer room as the resolved model allows — they cannot name
|
|
a number before knowing which model they got. The resolved value comes back as
|
|
`outputTokens`, which is what the caller should then request, so the reserve
|
|
and the request are the same number by construction.
|
|
- Reasoning models can spend the entire output budget thinking and return
|
|
nothing. That case (empty content with `finish_reason: 'length'`, or content
|
|
empty while `reasoning_content` is populated) throws with
|
|
`code: 'output-exhausted'` so callers can offer a different model instead of
|
|
showing a transport error.
|
|
- `timeoutMs` overrides the 60s default per call; `signal` lets a caller abort
|
|
a request that is no longer wanted. Both apply to every wire format.
|
|
- `describeSmallModel()` additionally reports `inputCharBudget`,
|
|
`contextTokens`, `contextKnown`, `structuredOutput`, and `hasLogin`. The last
|
|
is whether the resolved provider has a usable credential (`auth.json` or
|
|
config `provider.<id>.options.apiKey`) — settings/config overrides can name a
|
|
provider with none, and callers such as the walkthrough refuse before the
|
|
request. `structuredOutput` is tri-state: `true`/`false` from the catalog,
|
|
`null` when the catalog omits the field — which it does for roughly half of
|
|
all models, aggregators and proxies especially. Callers must treat `null` as
|
|
"try it", not "unsupported".
|
|
- Missing credentials throw with `statusCode: 401` and
|
|
`code: 'no-provider-login'` rather than a bare `Error`, so UI callers can show
|
|
a blocker instead of a raw 500 message.
|
|
- `call.js` — wire formats and per-provider auth, replicating OpenCode's
|
|
plugin auth loaders:
|
|
- **GitHub Copilot**: fetches the requested model's authenticated `/models`
|
|
metadata from `https://api.githubcopilot.com` (or
|
|
`copilot-api.<enterprise>`) and honors its advertised endpoint, preferring
|
|
Anthropic-compatible `/v1/messages`, then OpenAI `/responses`, then
|
|
`/chat/completions`. Models without `supported_endpoints` retain the legacy
|
|
Chat Completions default; metadata, missing-model, and unsupported-endpoint
|
|
failures are surfaced instead of guessing. The stored device-OAuth token is
|
|
used as the bearer with no token exchange or expiry.
|
|
- **OpenAI OAuth (ChatGPT plan)**: streaming Responses API on
|
|
`https://chatgpt.com/backend-api/codex/responses` with
|
|
`ChatGPT-Account-Id`; expired tokens are refreshed against
|
|
`auth.openai.com` (single-flight) and written back to `auth.json`.
|
|
- **Anthropic** (`type: api`): `/v1/messages` with `x-api-key`.
|
|
- **Google** (`type: api`): `generateContent` with `x-goog-api-key`; Gemini 3
|
|
uses `thinkingLevel` while older Flash models use `thinkingBudget: 0`.
|
|
- Everything else: OpenAI-compatible `/chat/completions` against the
|
|
provider's base URL, resolved from (1) `provider.<id>.options.baseURL`
|
|
in the OpenCode config, (2) the hardcoded `https://api.openai.com/v1`
|
|
endpoint, or (3) the provider's `api` field from the models.dev catalog.
|
|
Configured API keys honor OpenCode's `{env:NAME}` and `{file:path}`
|
|
substitutions; file contents and resolved credentials remain server-side.
|
|
- `[small-model:diagnostic]` logs record provider/model, input character
|
|
counts, output budget, thinking toggle, HTTP/finish status, and
|
|
content/reasoning lengths without logging prompts, response text, or
|
|
credentials. Goal audit parsing similarly emits
|
|
`[session-goal:diagnostic]` structural verdict metadata.
|
|
- `catalog.js` — models.dev catalog via the shared in-process cache
|
|
(`../opencode/models-metadata.js`, also serving
|
|
`/api/openchamber/models-metadata`).
|
|
- `routes.js` — `GET /api/small-model` (resolution preview) and
|
|
`POST /api/small-model/generate` (`{ prompt, system?, maxOutputTokens?,
|
|
model?, directory? }` → `{ text, providerID, modelID, source }`).
|
|
- `config-injection.js` — applies the Settings → Chat → Small Model override
|
|
to the config injected into the **managed OpenCode process**
|
|
(`OPENCODE_CONFIG_CONTENT`), so OpenCode's own internal `small_model`
|
|
consumers — session title and summary generation — use the user's explicit
|
|
choice instead of OpenCode's fallback chain. Only an explicit override
|
|
(`smallModelUseDefault === false` with a non-empty `smallModelOverride`) is
|
|
injected; "use default" leaves the config untouched so OpenCode's own
|
|
resolution stays authoritative. Wired into `getManagedOpenCodeEnv` in
|
|
`server/index.js`; the pure helper is unit-tested in
|
|
`config-injection.test.js`. External OpenCode servers are unaffected (they
|
|
are not launched with this env).
|
|
|
|
## Registration
|
|
|
|
Mounted lazily from `feature-routes-runtime.js` (same pattern as quota): the
|
|
module is imported on first request, not at server startup.
|
|
|
|
## Known limitations
|
|
|
|
- OpenCode's free models (`opencode/big-pickle`, `*-free`) work without a
|
|
token only through OpenCode's own server — direct calls are rejected, and
|
|
piggybacking on their subsidized infra is out of bounds by design. Every
|
|
resolution step therefore requires a usable auth entry for the provider:
|
|
a session on an unauthenticated `opencode` provider falls through to the
|
|
global scan (or a clean 404 on a vanilla setup with no logins).
|
|
|
|
- Anthropic OAuth (Claude Pro/Max) entries are not supported — OpenCode itself
|
|
keeps those outside `auth.json` in this generation; only `type: api` keys
|
|
work for Anthropic.
|
|
- Amazon Bedrock, GitLab, Azure and other credential-chain providers are out
|
|
of scope; they need more than a key/token (regions, resource names).
|
|
- Responses from the codex backend are collected from the SSE stream; the
|
|
endpoint itself is non-streaming by design (small utility calls).
|