openchamber/packages/web/server/lib/small-model/DOCUMENTATION.md

# Small Model

Server-side direct LLM calls that reuse the user's existing OpenCode provider
logins (`~/.local/share/opencode/auth.json`). OpenCode uses a "small model"
internally (titles, summaries) but does not expose it through the SDK or
plugins — this module replicates that mechanism as an OpenChamber runtime API.

## Security boundary

Credentials never leave the server process. The client sends only a prompt;
auth resolution, OAuth refresh, and provider dispatch all happen server-side.
Routes live under `/api/*` and are gated by the ui-auth middleware like every
other runtime API.

## Files

- `index.js` — orchestration: `generateSmallModelText()` / `describeSmallModel()`.
- `resolve.js` — model selection, mirroring OpenCode's `getSmallModel` chain:
  0. OpenChamber's own settings override (Settings → Sessions → Small Model):
     when `smallModelUseDefault` is `false`, `smallModelOverride`
     (`provider/model`) outranks everything below. Sanitized in
     `settings-helpers.js` (server), `persistence.ts` (client), and
     `bridge-settings-runtime.ts` (VS Code).
  1. `small_model` from the merged OpenCode config layers (`provider/model`).
  2. Family-priority scan (`gemini-flash` → `gpt-nano` → `claude-haiku`)
     **within the session's provider first** (`preferredProviderID`, like
     OpenCode resolves within the current provider), then over the other
     providers with a usable auth entry, newest `release_date` first.
  3. GitHub Copilot hidden utility models (`gpt-*-nano/mini`) — these never
     appear in the catalog, so they participate as the `gpt-nano` family entry
     and as a final utility fallback.
  4. Last resort: the session's own model (`preferredModelID`) when no small
     model resolves anywhere — costlier, but always valid.
- Input clamp: the prompt is measured against the resolved model's catalog
  `limit.context` (minus an output reserve, ~4 chars/token estimate;
  conservative default when the model is not in the catalog). `onOverflow`
  decides what an oversized prompt means:
  - `truncate` (default) clips the tail and reports `inputTruncated: true`.
    Correct for callers that degrade gracefully (summaries, commit messages).
  - `error` throws a `413` with `code: 'context-too-small'` plus
    `requiredChars`/`availableChars`. Correct for callers whose output would be
    quietly wrong on a clipped input, so they can ask the user for a roomier
    model instead of returning confident nonsense.
- Structured output: pass `responseSchema` (a JSON Schema) to get
  schema-shaped JSON back as `text`. Wire support differs per format —
  `response_format: {type: 'json_schema'}` for OpenAI-compatible chat,
  `text.format` for the Responses API, a forced single tool call for the
  Anthropic messages API, and `generationConfig.responseSchema` for Google
  (whose OpenAPI-flavored dialect drops unknown JSON Schema keywords). The
  ChatGPT-plan codex backend has no equivalent and rejects a schema request
  with `code: 'structured-output-unsupported'` rather than silently returning
  prose.
- Output budget: `maxOutputTokens` is capped at the catalog's `limit.output` for
  the model, and the **same number** is reserved from the input allowance. The
  two must not drift — a caller that asks for a large answer while the reserve
  stays at the default overruns the context, and the failure looks like a
  truncation bug rather than a budgeting one. `describeSmallModel` takes
  `outputReserveTokens` so readiness checks agree with what generation will do.
  It may be a **function** of `{ contextTokens, outputTokenLimit }` for callers
  that want as much answer room as the resolved model allows — they cannot name
  a number before knowing which model they got. The resolved value comes back as
  `outputTokens`, which is what the caller should then request, so the reserve
  and the request are the same number by construction.
- Reasoning models can spend the entire output budget thinking and return
  nothing. That case (empty content with `finish_reason: 'length'`, or content
  empty while `reasoning_content` is populated) throws with
  `code: 'output-exhausted'` so callers can offer a different model instead of
  showing a transport error.
- `timeoutMs` overrides the 60s default per call; `signal` lets a caller abort
  a request that is no longer wanted. Both apply to every wire format.
- `describeSmallModel()` additionally reports `inputCharBudget`,
  `contextTokens`, `contextKnown`, `structuredOutput`, and `hasLogin`. The last
  is whether the resolved provider has a usable credential (`auth.json` or
  config `provider.<id>.options.apiKey`) — settings/config overrides can name a
  provider with none, and callers such as the walkthrough refuse before the
  request. `structuredOutput` is tri-state: `true`/`false` from the catalog,
  `null` when the catalog omits the field — which it does for roughly half of
  all models, aggregators and proxies especially. Callers must treat `null` as
  "try it", not "unsupported".
- Missing credentials throw with `statusCode: 401` and
  `code: 'no-provider-login'` rather than a bare `Error`, so UI callers can show
  a blocker instead of a raw 500 message.
- `call.js` — wire formats and per-provider auth, replicating OpenCode's
  plugin auth loaders:
  - **GitHub Copilot**: fetches the requested model's authenticated `/models`
    metadata from `https://api.githubcopilot.com` (or
    `copilot-api.<enterprise>`) and honors its advertised endpoint, preferring
    Anthropic-compatible `/v1/messages`, then OpenAI `/responses`, then
    `/chat/completions`. Models without `supported_endpoints` retain the legacy
    Chat Completions default; metadata, missing-model, and unsupported-endpoint
    failures are surfaced instead of guessing. The stored device-OAuth token is
    used as the bearer with no token exchange or expiry.
  - **OpenAI OAuth (ChatGPT plan)**: streaming Responses API on
    `https://chatgpt.com/backend-api/codex/responses` with
    `ChatGPT-Account-Id`; expired tokens are refreshed against
    `auth.openai.com` (single-flight) and written back to `auth.json`.
  - **Anthropic** (`type: api`): `/v1/messages` with `x-api-key`.
  - **Google** (`type: api`): `generateContent` with `x-goog-api-key`; Gemini 3
    uses `thinkingLevel` while older Flash models use `thinkingBudget: 0`.
  - Everything else: OpenAI-compatible `/chat/completions` against the
    provider's base URL, resolved from (1) `provider.<id>.options.baseURL`
    in the OpenCode config, (2) the hardcoded `https://api.openai.com/v1`
     endpoint, or (3) the provider's `api` field from the models.dev catalog.
    Configured API keys honor OpenCode's `{env:NAME}` and `{file:path}`
    substitutions; file contents and resolved credentials remain server-side.
  - `[small-model:diagnostic]` logs record provider/model, input character
    counts, output budget, thinking toggle, HTTP/finish status, and
    content/reasoning lengths without logging prompts, response text, or
    credentials. Goal audit parsing similarly emits
    `[session-goal:diagnostic]` structural verdict metadata.
- `catalog.js` — models.dev catalog via the shared in-process cache
  (`../opencode/models-metadata.js`, also serving
  `/api/openchamber/models-metadata`).
- `routes.js` — `GET /api/small-model` (resolution preview) and
  `POST /api/small-model/generate` (`{ prompt, system?, maxOutputTokens?,
  model?, directory? }` → `{ text, providerID, modelID, source }`).

## Registration

Mounted lazily from `feature-routes-runtime.js` (same pattern as quota): the
module is imported on first request, not at server startup.

## Known limitations

- OpenCode's free models (`opencode/big-pickle`, `*-free`) work without a
  token only through OpenCode's own server — direct calls are rejected, and
  piggybacking on their subsidized infra is out of bounds by design. Every
  resolution step therefore requires a usable auth entry for the provider:
  a session on an unauthenticated `opencode` provider falls through to the
  global scan (or a clean 404 on a vanilla setup with no logins).

- Anthropic OAuth (Claude Pro/Max) entries are not supported — OpenCode itself
  keeps those outside `auth.json` in this generation; only `type: api` keys
  work for Anthropic.
- Amazon Bedrock, GitLab, Azure and other credential-chain providers are out
  of scope; they need more than a key/token (regions, resource names).
- Responses from the codex backend are collected from the SSE stream; the
  endpoint itself is non-streaming by design (small utility calls).