Dmitrii
3903cb53b1
readProviderConfig read only options.baseURL and options.apiKey, and the
OpenAI-compatible dispatch hardcoded a bearer token, so provider
options.headers never reached the request. OpenCode sends those headers on
every chat turn, which left the small model authenticating differently from
the request path against the same URL.
Providers behind a gateway that authenticates on its own header, such as the
Ocp-Apim-Subscription-Key default of Azure API Management, answered 401 for
walkthroughs, session goal audits, titles and commit summaries while the same
model worked in chat.
Read options.headers alongside the API key, resolve {env:...} and {file:...}
in the values with the existing resolveConfigApiKey, and merge them into the
request after the bearer default so a gateway whose header is the credential
can override it.
Closes #3213
224 lines
14 KiB
Markdown
224 lines
14 KiB
Markdown
# Small Model
|
|
|
|
Server-side direct LLM calls that reuse the user's existing OpenCode provider
|
|
logins (`~/.local/share/opencode/auth.json`). OpenCode uses a "small model"
|
|
internally (titles, summaries) but does not expose it through the SDK or
|
|
plugins — this module replicates that mechanism as an OpenChamber runtime API.
|
|
|
|
## Security boundary
|
|
|
|
Credentials never leave the server process. The client sends only a prompt;
|
|
auth resolution, OAuth refresh, and provider dispatch all happen server-side.
|
|
Routes live under `/api/*` and are gated by the ui-auth middleware like every
|
|
other runtime API.
|
|
|
|
## Files
|
|
|
|
- `index.js` — orchestration: `generateSmallModelText()` / `describeSmallModel()`.
|
|
- `runtime-providers.js` — provider state that exists only inside the running
|
|
OpenCode process. A plugin registers its provider from the `config` hook and
|
|
supplies the credential from its `auth` loader, so neither reaches
|
|
`opencode.json` nor `auth.json`; `GET /provider` is the only place they
|
|
become visible. The module caches one snapshot (30s TTL, shared in-flight
|
|
request) and answers `null` — never an empty provider list — when OpenCode is
|
|
unreachable, so a momentary outage cannot retract providers. It is wired once
|
|
from `server/index.js` and reset on OpenCode restart, which reloads plugins
|
|
and can move their ports and keys.
|
|
- `resolve.js` — model selection, mirroring OpenCode's `getSmallModel` chain:
|
|
0. OpenChamber's own settings override (Settings → Sessions → Small Model):
|
|
when `smallModelUseDefault` is `false`, `smallModelOverride`
|
|
(`provider/model`) outranks everything below. Sanitized in
|
|
`settings-helpers.js` (server), `persistence.ts` (client), and
|
|
`bridge-settings-runtime.ts` (VS Code).
|
|
1. `small_model` from the merged OpenCode config layers (`provider/model`).
|
|
2. Family-priority scan (`gemini-flash` → `gpt-nano` → `claude-haiku`)
|
|
**within the session's provider first** (`preferredProviderID`, like
|
|
OpenCode resolves within the current provider), then over the other
|
|
providers with a usable auth entry, newest `release_date` first.
|
|
3. GitHub Copilot hidden utility models (`gpt-*-nano/mini`) — these never
|
|
appear in the catalog, so they participate as the `gpt-nano` family entry
|
|
and as a final utility fallback.
|
|
4. Last resort: the session's own model (`preferredModelID`) when no small
|
|
model resolves anywhere — costlier, but always valid.
|
|
- Input clamp: the prompt is measured against the resolved model's catalog
|
|
`limit.context` (minus an output reserve, ~4 chars/token estimate;
|
|
conservative default when the model is not in the catalog). `onOverflow`
|
|
decides what an oversized prompt means:
|
|
- `truncate` (default) clips the tail and reports `inputTruncated: true`.
|
|
Correct for callers that degrade gracefully (summaries, commit messages).
|
|
- `error` throws a `413` with `code: 'context-too-small'` plus
|
|
`requiredChars`/`availableChars`. Correct for callers whose output would be
|
|
quietly wrong on a clipped input, so they can ask the user for a roomier
|
|
model instead of returning confident nonsense.
|
|
- Structured output: pass `responseSchema` (a JSON Schema) to get
|
|
schema-shaped JSON back as `text`. Wire support differs per format —
|
|
`response_format: {type: 'json_schema'}` for OpenAI-compatible chat,
|
|
`text.format` for the Responses API, a forced single tool call for the
|
|
Anthropic messages API, and `generationConfig.responseSchema` for Google
|
|
(whose OpenAPI-flavored dialect drops unknown JSON Schema keywords). The
|
|
ChatGPT-plan codex backend has no equivalent and rejects a schema request
|
|
with `code: 'structured-output-unsupported'` rather than silently returning
|
|
prose.
|
|
- Output budget: `maxOutputTokens` is capped at the catalog's `limit.output` for
|
|
the model, and the **same number** is reserved from the input allowance. The
|
|
two must not drift — a caller that asks for a large answer while the reserve
|
|
stays at the default overruns the context, and the failure looks like a
|
|
truncation bug rather than a budgeting one. `describeSmallModel` takes
|
|
`outputReserveTokens` so readiness checks agree with what generation will do.
|
|
It may be a **function** of `{ contextTokens, outputTokenLimit }` for callers
|
|
that want as much answer room as the resolved model allows — they cannot name
|
|
a number before knowing which model they got. The resolved value comes back as
|
|
`outputTokens`, which is what the caller should then request, so the reserve
|
|
and the request are the same number by construction.
|
|
- Reasoning models can spend the entire output budget thinking and return
|
|
nothing. That case (empty content with `finish_reason: 'length'`, or content
|
|
empty while `reasoning_content` is populated) throws with
|
|
`code: 'output-exhausted'` so callers can offer a different model instead of
|
|
showing a transport error.
|
|
- `timeoutMs` overrides the 60s default per call; `signal` lets a caller abort
|
|
a request that is no longer wanted. Both apply to every wire format.
|
|
- `describeSmallModel()` additionally reports `inputCharBudget`,
|
|
`contextTokens`, `contextKnown`, `structuredOutput`, and `hasLogin`. The last
|
|
is whether the resolved provider has a usable credential (`auth.json` or
|
|
config `provider.<id>.options.apiKey`) — settings/config overrides can name a
|
|
provider with none, and callers such as the walkthrough refuse before the
|
|
request. `structuredOutput` is tri-state: `true`/`false` from the catalog,
|
|
`null` when the catalog omits the field — which it does for roughly half of
|
|
all models, aggregators and proxies especially. Callers must treat `null` as
|
|
"try it", not "unsupported".
|
|
- Missing credentials throw with `statusCode: 401` and
|
|
`code: 'no-provider-login'` rather than a bare `Error`, so UI callers can show
|
|
a blocker instead of a raw 500 message.
|
|
- `call.js` — wire formats and per-provider auth, replicating OpenCode's
|
|
plugin auth loaders:
|
|
- **GitHub Copilot**: fetches the requested model's authenticated `/models`
|
|
metadata from `https://api.githubcopilot.com` (or
|
|
`copilot-api.<enterprise>`) and honors its advertised endpoint, preferring
|
|
Anthropic-compatible `/v1/messages`, then OpenAI `/responses`, then
|
|
`/chat/completions`. Models without `supported_endpoints` retain the legacy
|
|
Chat Completions default; metadata, missing-model, and unsupported-endpoint
|
|
failures are surfaced instead of guessing. The stored device-OAuth token is
|
|
used as the bearer with no token exchange or expiry.
|
|
- **OpenAI OAuth (ChatGPT plan)**: streaming Responses API on
|
|
`https://chatgpt.com/backend-api/codex/responses` with
|
|
`ChatGPT-Account-Id`; expired tokens are refreshed against
|
|
`auth.openai.com` (single-flight) and written back to `auth.json`.
|
|
- **Anthropic** (`type: api`): `/messages` with `x-api-key`, against
|
|
`provider.anthropic.options.baseURL` when configured (used as-is, matching
|
|
`@ai-sdk/anthropic` — no `/v1` is inserted) or `https://api.anthropic.com/v1`
|
|
otherwise.
|
|
- **Google** (`type: api`): `generateContent` with `x-goog-api-key`; Gemini 3
|
|
uses `thinkingLevel`, Gemini 2.x uses `thinkingBudget: 0`, and all other
|
|
models omit `thinkingConfig` entirely.
|
|
- Everything else: OpenAI-compatible `/chat/completions` against the
|
|
provider's base URL, resolved from (1) `provider.<id>.options.baseURL`
|
|
in the OpenCode config, (2) the hardcoded `https://api.openai.com/v1`
|
|
endpoint, (3) the endpoint OpenCode resolved at runtime, or (4) the
|
|
provider's `api` field from the models.dev catalog. The credential follows
|
|
the same shape: config `options.apiKey`, then the runtime credential, then
|
|
the auth.json entry. `provider.<id>.options.headers` is sent with the
|
|
request and overrides the bearer default, so gateways that authenticate on
|
|
their own header work here exactly as they do in a chat turn. Configured API
|
|
keys and header values honor OpenCode's `{env:NAME}` and `{file:path}`
|
|
substitutions; file contents and resolved credentials remain server-side.
|
|
- The runtime credential is refused for providers listed in
|
|
`OWN_CREDENTIAL_HANDLING`. Their branches need the stored entry rather than
|
|
a bearer token: the clearest case is the ChatGPT-plan `openai` login, whose
|
|
runtime `options.apiKey` is an OAuth access token that `api.openai.com`
|
|
answers with 401.
|
|
- `[small-model:diagnostic]` logs record provider/model, input character
|
|
counts, output budget, thinking toggle, HTTP/finish status, and
|
|
content/reasoning lengths without logging prompts, response text, or
|
|
credentials. Goal audit parsing similarly emits
|
|
`[session-goal:diagnostic]` structural verdict metadata.
|
|
- `catalog.js` — models.dev catalog via the shared in-process cache
|
|
(`../opencode/models-metadata.js`, also serving
|
|
`/api/openchamber/models-metadata`).
|
|
- `routes.js` — `GET /api/small-model` (resolution preview) and
|
|
`POST /api/small-model/generate` (`{ prompt, system?, maxOutputTokens?,
|
|
model?, directory? }` → `{ text, providerID, modelID, source }`).
|
|
- `config-injection.js` — applies the Settings → Chat → Small Model override
|
|
to the config injected into the **managed OpenCode process**
|
|
(`OPENCODE_CONFIG_CONTENT`), so OpenCode's own internal `small_model`
|
|
consumers — session title and summary generation — use the user's explicit
|
|
choice instead of OpenCode's fallback chain. Only an explicit override
|
|
(`smallModelUseDefault === false` with a non-empty `smallModelOverride`) is
|
|
injected; "use default" leaves the config untouched so OpenCode's own
|
|
resolution stays authoritative. Wired into `getManagedOpenCodeEnv` in
|
|
`server/index.js`; the pure helper is unit-tested in
|
|
`config-injection.test.js`. External OpenCode servers are unaffected (they
|
|
are not launched with this env).
|
|
|
|
## Which providers the pickers may offer
|
|
|
|
`listAuthenticatedProviders()` answers one question for the Small Model and
|
|
Changes Walkthrough pickers alike: which providers can this module actually
|
|
call. One rule decides it, applied the same way to every provider — **a
|
|
credential we are allowed to use, and an endpoint to send it to.** The
|
|
auth.json scan as before, plus the credential and endpoint OpenCode resolved
|
|
for a plugin provider.
|
|
|
|
**opencode zen is excluded without a real login.** When the user has no zen
|
|
credential, OpenCode substitutes the sentinel `options.apiKey = "public"` and
|
|
trims its catalog to the free models. Those run on OpenCode's own subsidised
|
|
infrastructure and are meant to be reached through OpenCode, so the sentinel is
|
|
never accepted as a credential — see `ZEN_ANONYMOUS_API_KEY`.
|
|
|
|
### Why there is no capability probe
|
|
|
|
A plugin may implement its whole integration inside `options.fetch` —
|
|
rewriting the path, signing the request, translating the payload — and OpenCode
|
|
cannot serialise a function. Such a provider advertises an ordinary base URL
|
|
that answers nothing we know how to ask, and no reported field distinguishes it
|
|
from a plain one.
|
|
|
|
Asking the endpoint (`GET /models`) does identify that case correctly. It was
|
|
measured against all 166 providers carrying an `api` URL in the models.dev
|
|
catalog, and it also denies six of them — `cloudflare-workers-ai`,
|
|
`infomaniak`, `iflowcn`, `inference`, `kuae-cloud-coding-plan`,
|
|
`thinkingmachines` — which work fine and simply have no `/models` route. At a
|
|
3.6% false-negative rate on providers known to work, the probe removes more
|
|
working models from the picker than broken ones, and a provider that silently
|
|
vanishes explains nothing while one that fails on use says why.
|
|
|
|
So availability stops at credential and endpoint, and the protocol verdict is
|
|
left to the call. A provider whose protocol lives in a plugin's `fetch` stays
|
|
selectable and fails when used — which is what it did before this resolution
|
|
existed.
|
|
|
|
Claude Code is refused unconditionally. A plugin can publish an
|
|
OpenAI-compatible endpoint for it, but that endpoint is a façade over the
|
|
Claude Agent SDK, which spawns the Claude Code CLI per request and spends the
|
|
user's Claude subscription rate limit. Paying that for a session title or a
|
|
summary is the wrong trade, so an available endpoint does not lift the
|
|
refusal — the cost is the reason, not the transport.
|
|
|
|
The result is served as `authenticatedProviders` on `GET /api/small-model`.
|
|
The field name predates the runtime resolution; it now means "callable", which
|
|
is a superset of "has an auth.json entry".
|
|
|
|
## Registration
|
|
|
|
Mounted lazily from `feature-routes-runtime.js` (same pattern as quota): the
|
|
module is imported on first request, not at server startup.
|
|
|
|
## Known limitations
|
|
|
|
- OpenCode's free models (`opencode/big-pickle`, `*-free`) work without a
|
|
token only through OpenCode's own server — direct calls are rejected, and
|
|
piggybacking on their subsidized infra is out of bounds by design. Every
|
|
resolution step therefore requires a credential we are allowed to use:
|
|
a session on an unauthenticated `opencode` provider falls through to the
|
|
global scan (or a clean 404 on a vanilla setup with no logins). The runtime
|
|
snapshot does not weaken this — OpenCode reports the sentinel
|
|
`apiKey: "public"` for that state, and this module refuses to read it as a
|
|
credential.
|
|
|
|
- Anthropic OAuth (Claude Pro/Max) entries are not supported — OpenCode itself
|
|
keeps those outside `auth.json` in this generation; only `type: api` keys
|
|
work for Anthropic.
|
|
- Amazon Bedrock, GitLab, Azure and other credential-chain providers are out
|
|
of scope; they need more than a key/token (regions, resource names).
|
|
- Responses from the codex backend are collected from the SSE stream; the
|
|
endpoint itself is non-streaming by design (small utility calls).
|