fix: resilient reconnect — preserve state on fetch fail, pause when offline (#1308)

* fix: preserve state when reconnect-time fetches fail

Several client API methods swallowed fetch/SDK errors and returned an
empty value (`[]`, `{}`), which was indistinguishable from a successful
"server says nothing here" response. Reconnect resync paths trusted that
empty result as authoritative and deleted local state — so after a
network blip (sleep/wake, wifi reconnect, tunnel switch), the UI could
show:

- sessions stuck on the "running" indicator (status never cleared)
- pending permission prompts disappearing from the UI
- pending question prompts disappearing from the UI

and only a page reload would recover. A related case: `listAgents`
silently returning `[]` defeated the 3-attempt retry loop in
`useAgentsStore` because the loop never saw an error.

The systematic fix:

- `getSessionStatusForDirectory` now returns `null` on fetch failure
  (vs the previous `{}`); the reconnect resync treats only a non-null
  response as authoritative — candidates missing from the response are
  written as `{type: "idle"}`, candidates after a failure are left
  untouched.
- `listPendingPermissions`, `listPendingQuestions`, and `listAgents`
  now throw on SDK/network failure. The pre-existing outer try/catch
  blocks in `resyncBlockingRequestsForDirectory` and the retry loop in
  `useAgentsStore` were already in the right shape — they just never
  fired because no exception was thrown. A small `formatSdkError`
  helper renders the SDK `{data, error}` shape into the thrown message.
- `permissionStore.setSessionAutoAccept` catches the new throw and
  falls back to whatever sync-store snapshots provide; the next SSE
  event or reconnect resync will catch up anything missed.

AGENTS.md gets a new "Distinguish fetch failure from empty success"
subsection documenting the principle (throw vs `T | null` patterns,
when to pick which, the retry-loop trap) so this doesn't regress.

Adds 3 regression tests covering the resync paths: existing
questions/permissions are preserved when the corresponding `list*`
method throws, and a permission-fetch failure does not block the
question block from running (verifies per-block try/catch isolation).

* fix: pause reconnect loop when offline or hidden

The SSE/WebSocket reconnect loop retried indefinitely with no awareness
of whether the browser was online or whether the tab was even visible.
Three issues compounded:

- No `online`/`offline` event handling. With a foreground tab on a dead
  network, we'd hit the server every ~5s forever, and on network
  recovery we'd wait up to ~5s for the next probe instead of reacting
  to the `online` event.
- No visibility awareness. A backgrounded PWA on a flaky link kept
  probing at the same rate as a foreground tab. The browser does
  throttle hidden-tab timers, but the intent wasn't expressed in code.
- The "exponential backoff" math
  `min(5000, max(retryDelayMs, 250) * (failures <= 1 ? 1 : 2))`
  re-initialized `retryDelayMs` to 250 every iteration, so the cap of
  5s was never reached — we waited 500ms forever after the second
  failure. Not actually exponential.

Now:

- `online` event aborts the current attempt (if disconnected) and
  cuts inter-attempt waits short. `offline` event aborts so the loop
  enters the slow-probe path immediately.
- `computeRetryDelay` returns the long cap (60s) when `navigator.onLine`
  is false or the tab is hidden; the short cap (5s) when foreground +
  online. The `online` event is the expected recovery path; the 60s cap
  is a fallback for browsers that miss the event.
- Real exponential growth: `BASE * 2^min(failures-1, 8)`, clamped.
- New `waitForRetry` helper interrupts on `online`,
  visibility-becomes-visible, and abort signal — so visibility/network
  recovery doesn't wait out the rest of the current sleep.

AGENTS.md gets a "Reconnect-loop pacing" subsection alongside the
fetch-failure rule, since they're the same family of resilience
concerns.

One regression test: simulates offline + failed first attempt + `online`
event after the failure; verifies the next attempt fires within seconds
instead of waiting the full 60s offline cap.

* fix: long-cap backoff for permanent 4xx server errors

Before this commit the reconnect loop didn't distinguish HTTP error
types. A stuck-path client (wrong URL after server upgrade) or an
expired-auth client (stale token) would hit the server at the normal
5-second cap forever — ~12 reqs/min, indefinitely, with no path to
recovery besides the user reloading.

Now the catch block extracts an HTTP status (looking on `error.status`
and `error.response.status` — the SDK exposes both depending on the
code path) and overrides the backoff:

- 4xx other than 408/429 → use the long cap (60s) immediately.
  Blind retries won't fix wrong path / bad auth / forbidden, so don't
  pound the server. waitForRetry's `online` / visibility-visible
  interrupters still apply — when an operator fixes the server-side
  config and the client comes back to foreground, recovery is prompt.
- 408 (Request Timeout) and 429 (Too Many Requests) → normal
  exponential path. Those are retryable in spirit.
- 5xx / network / unknown → normal exponential path. Unchanged.

AGENTS.md gets a new bullet under "Reconnect-loop pacing" covering
this — the rule fits naturally alongside the existing `navigator.onLine`
and visibility signals.

Two regression tests:
- A 404-throwing SDK doesn't fire a second attempt within 250ms (proves
  we left the exponential path). After `online` interrupts the wait,
  subsequent attempts fire promptly — proves the override doesn't break
  recovery once the underlying problem is fixed.
- A 429-throwing SDK recovers within 2s — proves 429 still hits the
  fast exponential path and isn't caught by the permanent-error branch.

---------

Co-authored-by: vhqtvn <8930337+vhqtvn@users.noreply.github.com>

packages/ui/src/sync/__tests__/event-pipeline-online.test.js

@@ -0,0 +1,107 @@
+import { afterEach, describe, expect, it } from 'bun:test';
+import { createEventPipeline } from '../event-pipeline';
+
+const savedDocument = globalThis.document;
+const savedWindow = globalThis.window;
+const savedNavigator = globalThis.navigator;
+
+afterEach(() => {
+  globalThis.document = savedDocument;
+  globalThis.window = savedWindow;
+  globalThis.navigator = savedNavigator;
+});
+
+// Multi-listener event-target stub. The simpler single-slot stub used in
+// event-pipeline-resume.test.js would break here because waitForRetry and the
+// top-level onOnline handler both register for `online`.
+function createEventTarget(extras = {}) {
+  const listeners = new Map();
+  return {
+    ...extras,
+    addEventListener(event, handler) {
+      const list = listeners.get(event);
+      if (list) list.add(handler);
+      else listeners.set(event, new Set([handler]));
+    },
+    removeEventListener(event, handler) {
+      listeners.get(event)?.delete(handler);
+    },
+    dispatch(event) {
+      const list = listeners.get(event);
+      if (!list) return;
+      for (const handler of Array.from(list)) {
+        handler();
+      }
+    },
+  };
+}
+
+describe('createEventPipeline — online event', () => {
+  it('cuts the inter-attempt wait short when `online` fires after disconnect', async () => {
+    globalThis.document = createEventTarget({ visibilityState: 'visible' });
+    globalThis.window = createEventTarget({
+      location: { href: 'http://127.0.0.1:3000/', origin: 'http://127.0.0.1:3000' },
+    });
+    globalThis.navigator = { onLine: false };
+
+    let sdkCallIndex = 0;
+    const sdk = {
+      global: {
+        event: async () => {
+          const idx = sdkCallIndex++;
+          if (idx === 0) {
+            // Force a real failure so the loop enters the offline backoff path
+            // (computeRetryDelay returns the long cap because navigator.onLine
+            // is false). Without our `online` interrupt this would wait the
+            // full hidden/offline cap of 60s and the test would time out.
+            throw new Error('simulated network error');
+          }
+          return {
+            stream: (async function* () {
+              yield {
+                payload: {
+                  type: 'session.status',
+                  properties: { sessionID: 's1', status: { type: 'idle' } },
+                },
+              };
+              await new Promise(() => {});
+            })(),
+          };
+        },
+      },
+    };
+
+    const startedAt = Date.now();
+    const elapsed = await new Promise((resolve) => {
+      let connects = 0;
+      const { cleanup } = createEventPipeline({
+        sdk,
+        transport: 'sse',
+        heartbeatTimeoutMs: 60_000,
+        reconnectDelayMs: 60_000,
+        onEvent: () => {},
+        onDisconnect: () => {
+          // We're now inside waitForRetry on the long offline cap.
+          // Flip the browser back online and fire the event; waitForRetry
+          // should resolve early and the next attempt should fire.
+          setTimeout(() => {
+            globalThis.navigator = { onLine: true };
+            globalThis.window.dispatch('online');
+          }, 30);
+        },
+        onReconnect: () => {
+          connects += 1;
+          if (connects === 1) {
+            cleanup();
+            resolve(Date.now() - startedAt);
+          }
+        },
+      });
+    });
+
+    // Two attempts: the failed one + the recovery one. If the `online`
+    // interrupt didn't fire, the test would have hung on the 60s offline cap.
+    expect(sdkCallIndex).toBe(2);
+    expect(elapsed).toBeLessThan(2_000);
+  });
+});

packages/ui/src/sync/__tests__/event-pipeline-permanent-error.test.js

@@ -0,0 +1,186 @@
+import { afterEach, describe, expect, it } from 'bun:test';
+import { createEventPipeline } from '../event-pipeline';
+
+const savedDocument = globalThis.document;
+const savedWindow = globalThis.window;
+const savedNavigator = globalThis.navigator;
+
+afterEach(() => {
+  globalThis.document = savedDocument;
+  globalThis.window = savedWindow;
+  globalThis.navigator = savedNavigator;
+});
+
+function createEventTarget(extras = {}) {
+  const listeners = new Map();
+  return {
+    ...extras,
+    addEventListener(event, handler) {
+      const list = listeners.get(event);
+      if (list) list.add(handler);
+      else listeners.set(event, new Set([handler]));
+    },
+    removeEventListener(event, handler) {
+      listeners.get(event)?.delete(handler);
+    },
+    dispatch(event) {
+      const list = listeners.get(event);
+      if (!list) return;
+      for (const handler of Array.from(list)) {
+        handler();
+      }
+    },
+  };
+}
+
+describe('createEventPipeline — permanent server errors', () => {
+  it('uses the long backoff cap for 4xx so we do not hammer at 5s intervals', async () => {
+    globalThis.document = createEventTarget({ visibilityState: 'visible' });
+    globalThis.window = createEventTarget({
+      location: { href: 'http://127.0.0.1:3000/', origin: 'http://127.0.0.1:3000' },
+    });
+    globalThis.navigator = { onLine: true };
+
+    let sdkCallIndex = 0;
+    const sdk = {
+      global: {
+        event: async () => {
+          const idx = sdkCallIndex++;
+          if (idx <= 1) {
+            // First two attempts: permanent 404. Under the old code these
+            // would have entered the exponential path and the second retry
+            // would fire after ~250-500ms. With the permanent-error override
+            // both go to the long (60s) cap, so the test should observe
+            // exactly one retry (after `online` interrupts) within its
+            // observation window.
+            const error = new Error('Not Found');
+            error.status = 404;
+            throw error;
+          }
+          return {
+            stream: (async function* () {
+              yield {
+                payload: {
+                  type: 'session.status',
+                  properties: { sessionID: 's1', status: { type: 'idle' } },
+                },
+              };
+              await new Promise(() => {});
+            })(),
+          };
+        },
+      },
+    };
+
+    const startedAt = Date.now();
+    let cleanupFn = () => {};
+
+    // Phase 1: let the first 404 fire and verify the loop is NOT spinning.
+    // If the permanent-error override is broken, the loop would retry every
+    // 250-500ms and sdkCallIndex would climb past 1.
+    await new Promise((resolve) => {
+      const { cleanup } = createEventPipeline({
+        sdk,
+        transport: 'sse',
+        heartbeatTimeoutMs: 60_000,
+        reconnectDelayMs: 60_000,
+        onEvent: () => {},
+        onDisconnect: () => {
+          // Wait 250ms after disconnect — long enough that the broken
+          // exponential path would have retried at least once. If our
+          // override works, sdkCallIndex stays at 1.
+          setTimeout(resolve, 250);
+        },
+      });
+      cleanupFn = cleanup;
+    });
+
+    expect(sdkCallIndex).toBe(1);
+
+    // Phase 2: fire `online` to interrupt the long wait. Loop should fire
+    // the second attempt (still 404) immediately, then the third attempt
+    // which succeeds.
+    const recovered = new Promise((resolve) => {
+      // Trigger an `online` event; waitForRetry's interrupter resolves and
+      // the next attempt fires. That attempt is also a 404 (idx=1), then
+      // another `online` advances us to the success path (idx=2).
+      const advance = () => {
+        globalThis.window.dispatch('online');
+      };
+      advance();
+      const t = setInterval(() => {
+        if (sdkCallIndex >= 3) {
+          clearInterval(t);
+          resolve();
+        } else {
+          advance();
+        }
+      }, 50);
+    });
+
+    await recovered;
+    cleanupFn();
+
+    expect(sdkCallIndex).toBeGreaterThanOrEqual(3);
+    // Total elapsed should be < 2s — well under the 60s cap that proves the
+    // interrupters work for permanent-error retries too.
+    expect(Date.now() - startedAt).toBeLessThan(5_000);
+  });
+
+  it('retries 408 and 429 on the normal exponential path (not the permanent cap)', async () => {
+    globalThis.document = createEventTarget({ visibilityState: 'visible' });
+    globalThis.window = createEventTarget({
+      location: { href: 'http://127.0.0.1:3000/', origin: 'http://127.0.0.1:3000' },
+    });
+    globalThis.navigator = { onLine: true };
+
+    let sdkCallIndex = 0;
+    const sdk = {
+      global: {
+        event: async () => {
+          const idx = sdkCallIndex++;
+          if (idx === 0) {
+            const error = new Error('Rate limited');
+            error.status = 429;
+            throw error;
+          }
+          return {
+            stream: (async function* () {
+              yield {
+                payload: {
+                  type: 'session.status',
+                  properties: { sessionID: 's1', status: { type: 'idle' } },
+                },
+              };
+              await new Promise(() => {});
+            })(),
+          };
+        },
+      },
+    };
+
+    const startedAt = Date.now();
+    const elapsed = await new Promise((resolve) => {
+      let connects = 0;
+      const { cleanup } = createEventPipeline({
+        sdk,
+        transport: 'sse',
+        heartbeatTimeoutMs: 60_000,
+        reconnectDelayMs: 60_000,
+        onEvent: () => {},
+        onReconnect: () => {
+          connects += 1;
+          if (connects === 1) {
+            cleanup();
+            resolve(Date.now() - startedAt);
+          }
+        },
+      });
+    });
+
+    // 429 went through computeRetryDelay (consecutiveFailures=1) -> 250ms,
+    // not the 60s permanent cap. Recovery should be sub-second.
+    expect(sdkCallIndex).toBe(2);
+    expect(elapsed).toBeLessThan(2_000);
+  });
+});

packages/ui/src/sync/__tests__/session-switch-resync.test.ts

@@ -6,15 +6,19 @@ const listPendingQuestionsCalls: Array<{ directories?: Array<string | null | und
 const listPendingPermissionsCalls: Array<{ directories?: Array<string | null | undefined> }> = []
 let pendingQuestionsResponse: QuestionRequest[] = []
 let pendingPermissionsResponse: PermissionRequest[] = []
+let pendingQuestionsShouldThrow = false
+let pendingPermissionsShouldThrow = false
 
 mock.module("@/lib/opencode/client", () => ({
   opencodeClient: {
     listPendingQuestions: mock(async (opts?: { directories?: Array<string | null | undefined> }) => {
       listPendingQuestionsCalls.push(opts ?? {})
+      if (pendingQuestionsShouldThrow) throw new Error("question.list failed: simulated")
       return pendingQuestionsResponse
     }),
     listPendingPermissions: mock(async (opts?: { directories?: Array<string | null | undefined> }) => {
       listPendingPermissionsCalls.push(opts ?? {})
+      if (pendingPermissionsShouldThrow) throw new Error("permission.list failed: simulated")
       return pendingPermissionsResponse
     }),
     getDirectory: () => "/repo",
@@ -85,6 +89,8 @@ describe("resyncBlockingRequestsForDirectory", () => {
     listPendingPermissionsCalls.length = 0
     pendingQuestionsResponse = []
     pendingPermissionsResponse = []
+    pendingQuestionsShouldThrow = false
+    pendingPermissionsShouldThrow = false
   })
 
   test("calls listPendingQuestions and listPendingPermissions exactly once for the directory", async () => {
@@ -156,4 +162,47 @@ describe("resyncBlockingRequestsForDirectory", () => {
     expect(listPendingQuestionsCalls).toHaveLength(0)
     expect(listPendingPermissionsCalls).toHaveLength(0)
   })
+
+  // Regression: prior to the fix, listPendingQuestions silently returned [] on
+  // fetch failure, indistinguishable from a successful empty server response.
+  // The resync then walked the candidate set and deleted any question that
+  // wasn't in the (empty) result — wiping legitimate in-flight prompts on a
+  // transient network blip. The client method now throws on failure and the
+  // outer try/catch preserves existing state.
+  test("preserves existing questions when listPendingQuestions throws (transient fetch failure)", async () => {
+    const store = createDirectoryStore({
+      question: { ses_a: [{ ...buildQuestion(), id: "que_in_flight" }] },
+    })
+    pendingQuestionsShouldThrow = true
+
+    await resyncBlockingRequestsForDirectory("/repo", store)
+
+    expect(store.getState().question["ses_a"]).toHaveLength(1)
+    expect(store.getState().question["ses_a"]?.[0]?.id).toBe("que_in_flight")
+  })
+
+  test("preserves existing permissions when listPendingPermissions throws (transient fetch failure)", async () => {
+    const store = createDirectoryStore({
+      permission: { ses_a: [{ ...buildPermission(), id: "perm_in_flight" }] },
+    })
+    pendingPermissionsShouldThrow = true
+
+    await resyncBlockingRequestsForDirectory("/repo", store)
+
+    expect(store.getState().permission["ses_a"]).toHaveLength(1)
+    expect(store.getState().permission["ses_a"]?.[0]?.id).toBe("perm_in_flight")
+  })
+
+  test("permission fetch failure does not block question resync (and vice versa)", async () => {
+    const store = createDirectoryStore({})
+    pendingQuestionsResponse = [buildQuestion()]
+    pendingPermissionsShouldThrow = true
+
+    await resyncBlockingRequestsForDirectory("/repo", store)
+
+    // Question block ran successfully despite permission block failing.
+    expect(store.getState().question["ses_a"]).toHaveLength(1)
+    expect(store.getState().question["ses_a"]?.[0]?.id).toBe("que_1")
+    expect(listPendingPermissionsCalls).toHaveLength(1)
+  })
 })