fix: add concurrency controls for multiple sessions using the same provider (#1069)

* fix: add concurrency controls for multiple sessions using the same provider

Adds OS-inspired scheduling primitives (from HiveMind/AIMD research) to prevent
concurrent sessions from the same provider from experiencing slowdowns, random
stops, and cascading failures.

Server-side:
- Health check skips OpenCode restart when sessions are actively busy — a busy
  server under concurrent load can fail the health check timeout without being
  dead. Staleness guard forces restart if unhealthy+busy persists >2 minutes.
- Upstream SSE stall timeout scaled from 20s to 60s to avoid unnecessary
  reconnections when multiple sessions are waiting for LLM responses.

Client-side (HiveMind primitives, arXiv:2604.17111):
- Transparent retry with exponential backoff (1s→2s→4s, max 32s) for
  429/502/503/504 errors — the #1 most effective primitive from the paper.
- Circuit breaker: opens after 3 consecutive retryable errors, cooldown
  doubles each trip (30s→60s→120s, capped 128s), matching TCP AIMD.
- Per-provider session tracking with TTL eviction (1h idle sweep).
- Fetch-level retry gated on AbortError/TypeError only (not DNS failures).

Refs github-code-review skill findings (all 8 issues resolved).

* fix: use definite assignment assertion for response variable

Fixes TS2454: Variable 'response' is used before being assigned
in strict mode. The for-loop body always assigns it on every path
that reaches the post-loop code, but TS can't prove that.

* fix: add cleanupSession to error paths and remove unreachable code

P1 fixes (Greptile review):
- cleanupSession called on fetch error throw path
- cleanupSession called on non-retryable HTTP error throw path
- Removed unreachable post-loop code (loop always terminates via return or throw)

Adds explicit post-loop throw to satisfy TypeScript strict return check.

* fix: address Greptile review feedback on concurrent session controls

Removes client-side session tracking that leaked on normal completion paths.

The session tracking was redundant — the server-side health check already reads from

sessionRuntime.getSessionActivitySnapshot() for busy-session detection.

Changes:

- Remove activeSessions Set and all session-tracking functions from provider-tracker

- Remove trackSessionStarted/cleanupSession calls from client.ts

- Remove unreachable (response as Response) block after retry loop

- Make upstreamStallTimeoutMs conditional: 60s when >1 sessions, 20s otherwise

Refs #1069

* fix: enforce dynamic concurrency safeguards

---------

Co-authored-by: Bohdan Triapitsyn <artmore@protonmail.com>

packages/web/server/lib/opencode/lifecycle.js

@@ -25,6 +25,7 @@ export const createOpenCodeLifecycleRuntime = (deps) => {
     buildAugmentedPath,
     buildManagedOpenCodePath,
     getManagedOpenCodeShellEnvSnapshot,
+    getActiveSessionCount = () => 0,
   } = deps;
 
   const killProcessOnPort = (port) => {
@@ -796,15 +797,51 @@ export const createOpenCodeLifecycleRuntime = (deps) => {
    * Perform an immediate (one-shot) health check and restart OpenCode if it's
    * not healthy.  Callers on the SSE / WS proxy path use this to trigger
    * recovery without waiting for the next periodic interval (up to 15 s).
+   *
+   * Skips restart when sessions are actively busy — a busy server under
+   * concurrent load can fail the health check timeout without actually
+   * being dead (the health endpoint competes with LLM work).
+   * Forces restart if sessions stay "busy" and the server stays unhealthy
+   * for over 2 minutes (staleness guard against stuck session state).
    */
+  const STALE_BUSY_GRACE_MS = 2 * 60 * 1000;
+  let lastUnhealthyWithBusySessionsAt = 0;
+
+  const shouldSkipRestartForBusySessions = () => {
+    const activeCount = getActiveSessionCount();
+    if (activeCount === 0) {
+      lastUnhealthyWithBusySessionsAt = 0;
+      return false;
+    }
+
+    const now = Date.now();
+    if (!lastUnhealthyWithBusySessionsAt) {
+      lastUnhealthyWithBusySessionsAt = now;
+      return true;
+    }
+
+    if (now - lastUnhealthyWithBusySessionsAt >= STALE_BUSY_GRACE_MS) {
+      console.warn(
+        `[lifecycle] OpenCode unhealthy with ${activeCount} busy session(s) for > 2 min — forcing restart`
+      );
+      lastUnhealthyWithBusySessionsAt = 0;
+      return false;
+    }
+
+    return true;
+  };
+
   const triggerHealthCheck = async () => {
     if (!state.openCodeProcess || state.isShuttingDown || state.isRestartingOpenCode) return;
 
     try {
       const healthy = await isOpenCodeProcessHealthy();
       if (!healthy) {
+        if (shouldSkipRestartForBusySessions()) return;
         console.log('[lifecycle] immediate health check: OpenCode not healthy, restarting...');
         await restartOpenCode();
+      } else {
+        lastUnhealthyWithBusySessionsAt = 0;
       }
     } catch (error) {
       console.error(`[lifecycle] immediate health check error: ${error.message}`);
@@ -822,8 +859,11 @@ export const createOpenCodeLifecycleRuntime = (deps) => {
       try {
         const healthy = await isOpenCodeProcessHealthy();
         if (!healthy) {
+          if (shouldSkipRestartForBusySessions()) return;
           console.log('OpenCode process not running, restarting...');
           await restartOpenCode();
+        } else {
+          lastUnhealthyWithBusySessionsAt = 0;
         }
       } catch (error) {
         console.error(`Health check error: ${error.message}`);

packages/web/server/lib/opencode/startup-pipeline-runtime.js

@@ -24,6 +24,7 @@ export const createStartupPipelineRuntime = (dependencies) => {
       processForwardedEventPayload,
       messageStreamWsClients,
       triggerHealthCheck,
+      upstreamStallTimeoutMs,
       terminalHeartbeatIntervalMs,
       terminalRebindWindowMs,
       terminalMaxRebindsPerWindow,
@@ -80,6 +81,7 @@ export const createStartupPipelineRuntime = (dependencies) => {
       processForwardedEventPayload,
       wsClients: messageStreamWsClients,
       triggerHealthCheck,
+      upstreamStallTimeoutMs,
     });
 
     setupProxy(app);