fix: settle busy sessions after managed OpenCode restart (#3002)

* fix: reconcile busy sessions after managed OpenCode restart

Forced health-check restarts previously rebound the event stream without
settling in-flight turns, so sessions stayed busy with no terminal state.
Interrupt those sessions, classify health failures, and retain bounded
process diagnostics for post-restart diagnosis.

Fixes #2943

Co-authored-by: serkraser <serkraser@gmail.com>

* fix: surface interrupted chats after OpenCode restart

Complete unfinished assistant turns as aborted once the session is
authoritatively idle, and show a persistent toast so users can continue
instead of remaining silently stranded.

Fixes #2943

Co-authored-by: serkraser <serkraser@gmail.com>

* fix: redact Basic auth credentials in restart diagnostics

The key/value sanitizer stopped at whitespace, so Authorization: Basic
credentials survived in stderr tails and health snapshots. Redact the
scheme token before that rule runs.

Co-authored-by: serkraser <serkraser@gmail.com>

packages/ui/src/sync/DOCUMENTATION.md

@@ -209,7 +209,7 @@ Incomplete-session materialization is deduplicated by runtime, directory, and se
 
 When `session.idle` or `session.error` settles a session but the trailing assistant message still contains a `pending` or `running` tool, sync refreshes that session tail. This narrowly reconciles a missed terminal tool-part event without refetching normally completed turns or stale tools from older turns. A stale refresh or delayed part event cannot regress a locally observed terminal tool to an active status.
 
-When a session is authoritatively settled — `session.idle`/`session.error` event, or an authoritative status snapshot that lowers a previously busy session — and the trailing assistant message is still *unfinished* (`time.completed` missing) with active tool parts and no pending question/permission, the turn is treated as interrupted (managed OpenCode process died mid-turn; the server never finalizes the parts, see openchamber#2577 / anomalyco/opencode#19023). The active parts are finalized locally as `error`/`Interrupted` with an end time, so tool timers stop and cards render the error state. The mark is gated on an explicit idle status (absent status is "unknown", never judged), never applies while the session is busy (including question/permission waits), and a later terminal event or refresh supersedes it while a stale `running` refresh cannot regress it.
+When a session is authoritatively settled — `session.idle`/`session.error` event, or an authoritative status snapshot that lowers a previously busy session — and the trailing assistant message is still *unfinished* (`time.completed` missing) with no pending question/permission, the turn is treated as interrupted (managed OpenCode process died mid-turn; the server never finalizes the message or parts, see openchamber#2577 / anomalyco/opencode#19023). The unfinished assistant message is completed locally with `MessageAbortedError`, including text-only turns and turns whose tools had already finished, so the chat shows a visible interrupted state. Any active parts are also finalized as `error`/`Interrupted` with an end time, so tool timers stop and cards render the error state. The mark is gated on an explicit idle status (absent status is "unknown", never judged), never applies while the session is busy (including question/permission waits), and a later terminal event can supersede it while a stale unfinished refresh cannot regress the locally finalized message or parts.
 
 Directory stores also own session-keyed sidecar notification channels for permissions, questions, and message materialization. High-frequency realtime part events annotate the exact session/message before committing, so visible records, user history, renderability, and sidebar permission and question rows are not notified by unrelated sessions. Structural message replacements notify only changed subscribed session buckets; unannotated bulk part replacement conservatively resets active message subscribers so bootstrap, pagination, rollback, and legacy writers cannot leave stale projections.
 

packages/ui/src/sync/__tests__/interrupted-turn-tools.test.ts

@@ -3,7 +3,7 @@
  * process dies mid-turn, the persisted turn never settles — the trailing
  * assistant message has no time.completed and its tool parts stay running.
  * Once the session is authoritatively settled, `interruptedTurnToolParts`
- * finalizes the orphaned parts locally.
+ * completes the assistant message as aborted and finalizes orphaned parts.
  */
 import { describe, expect, test } from "bun:test"
 import type { Message, Part } from "@opencode-ai/sdk/v2/client"
@@ -74,10 +74,15 @@ describe("interruptedTurnToolParts (#2577)", () => {
 
     const result = interruptedTurnToolParts(store, "ses_1", 5000)
     expect(result).not.toBeNull()
-    const part = result!.parts[0] as { state: { status: string; error: string; time: { end: number } } }
+    const part = result!.parts![0] as { state: { status: string; error: string; time: { end: number } } }
     expect(part.state.status).toBe("error")
     expect(part.state.error).toBe("Interrupted")
     expect(part.state.time.end).toBe(5000)
+    expect(result!.messages[0]).toEqual({
+      ...unfinishedAssistantMessage("msg_1"),
+      time: { created: 10, completed: 5000 },
+      error: { name: "MessageAbortedError", data: { message: "aborted" }, message: "aborted" },
+    })
   })
 
   test("busy session is never marked (live work)", () => {
@@ -137,16 +142,43 @@ describe("interruptedTurnToolParts (#2577)", () => {
 
     const result = interruptedTurnToolParts(store, "ses_1", 5000)
     expect(result).not.toBeNull()
-    const statuses = result!.parts.map((part) => (part as { state: { status: string } }).state.status)
+    const statuses = result!.parts!.map((part) => (part as { state: { status: string } }).state.status)
     expect(statuses).toEqual(["error", "completed", "error"])
   })
 
-  test("no active parts → no change", () => {
+  test("unfinished assistant with no tools is completed as aborted", () => {
     const store = state({
       session_status: { ses_1: { type: "idle" } },
       message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
-      part: { msg_1: [completedTool("tool_2", "msg_1")] },
+      part: {},
+    })
+
+    const result = interruptedTurnToolParts(store, "ses_1", 5000)
+    expect(result).not.toBeNull()
+    expect(result!.parts).toBe(undefined)
+    expect(result!.messages[0]).toEqual({
+      ...unfinishedAssistantMessage("msg_1"),
+      time: { created: 10, completed: 5000 },
+      error: { name: "MessageAbortedError", data: { message: "aborted" }, message: "aborted" },
+    })
+  })
+
+  test("completed tools are untouched while the unfinished assistant is aborted", () => {
+    const completed = completedTool("tool_2", "msg_1")
+    const store = state({
+      session_status: { ses_1: { type: "idle" } },
+      message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
+      part: { msg_1: [completed] },
+    })
+
+    const result = interruptedTurnToolParts(store, "ses_1", 5000)
+    expect(result).not.toBeNull()
+    expect(result!.parts).toBe(undefined)
+    expect(store.part.msg_1[0]).toBe(completed)
+    expect(result!.messages[0]).toEqual({
+      ...unfinishedAssistantMessage("msg_1"),
+      time: { created: 10, completed: 5000 },
+      error: { name: "MessageAbortedError", data: { message: "aborted" }, message: "aborted" },
     })
-    expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
   })
 })

packages/ui/src/sync/__tests__/materialization.test.ts

@@ -119,6 +119,31 @@ describe("materializeSessionSnapshots", () => {
     expect(result.part.msg_1[0]).toBe(livePart)
   })
 
+  test("preserves a locally aborted assistant message when a stale unfinished snapshot arrives", () => {
+    const unfinishedMessage = message("msg_1")
+    if (unfinishedMessage.role !== "assistant") throw new Error("Expected assistant fixture")
+    const abortedMessage: Message = {
+      ...unfinishedMessage,
+      time: { created: 1, completed: 5000 },
+      error: { name: "MessageAbortedError", data: { message: "aborted" } },
+    }
+    const staleMessage = message("msg_1")
+    const state = {
+      message: { ses_1: [abortedMessage] },
+      part: { msg_1: [] },
+    }
+
+    const result = materializeSessionSnapshots(
+      state,
+      "ses_1",
+      [{ info: staleMessage, parts: [] }],
+    )
+
+    expect(result.message).toBe(state.message)
+    expect(result.message.ses_1[0]).toBe(abortedMessage)
+    expect(result.message.ses_1[0]).not.toBe(staleMessage)
+  })
+
   test("does not preserve omitted optimistic user text parts beside server snapshot parts", () => {
     const optimisticPart = { id: "prt_optimistic", messageID: "msg_1", type: "text", text: "Hello" } as Part
     const serverPart = part("prt_server", "msg_1", "text", "Hello")

packages/ui/src/sync/sync-context.tsx

@@ -70,6 +70,7 @@ import { getRuntimeLiveStatusSeed, LIVE_STATUS_TTL_MS } from "./runtime-live-mem
 import { getRuntimeKey } from "@/lib/runtime-switch"
 import { getRegisteredRuntimeAPIs } from "@/contexts/runtimeAPIRegistry"
 import { isFilesystemError } from "@/lib/api/files-errors"
+import { formatMessage, useI18nStore } from "@/lib/i18n"
 import { listGlobalSessionPages } from "@/stores/globalSessions"
 import { areRequestArraysReferentiallyEqual, collectScopedBlockingRequests } from "./scoped-blocking-requests"
 import { EMPTY_USER_MESSAGE_HISTORY_SNAPSHOT, buildUserMessageHistorySnapshot, type UserMessageHistorySnapshot } from "./user-message-history"
@@ -456,6 +457,32 @@ const handleUiNotificationEvent = (payload: Event, fallbackDirectory: string): b
   }
 
   const notification = properties as UiNotificationPayload
+  const kind = asOptionalString(notification.kind)
+  const sessionId = asOptionalString(notification.sessionId)
+  const directory = asOptionalString(notification.directory)
+    ?? (fallbackDirectory !== "global" ? fallbackDirectory : "")
+
+  if (kind === "opencode-restart-interrupted") {
+    const dictionary = useI18nStore.getState().dictionary
+    const title = formatMessage(dictionary, "chat.toast.opencodeRestartInterrupted.title")
+    const options = {
+      id: "opencode-restart-interrupted",
+      description: formatMessage(dictionary, "chat.toast.opencodeRestartInterrupted.description"),
+      duration: Infinity,
+    }
+    if (sessionId && directory) {
+      toast.info(title, {
+        ...options,
+        action: {
+          label: formatMessage(dictionary, "chat.toast.opencodeRestartInterrupted.openSession"),
+          onClick: () => openSessionFromToast(sessionId, directory),
+        },
+      })
+    } else {
+      toast.info(title, options)
+    }
+  }
+
   if ((notification.desktopNotificationDelivered === true || notification.desktopStdoutActive === true) && getRuntimeKey() === "local") {
     return true
   }
@@ -469,9 +496,9 @@ const handleUiNotificationEvent = (payload: Event, fallbackDirectory: string): b
     title: asOptionalString(notification.title),
     body: asOptionalString(notification.body),
     tag: asOptionalString(notification.tag),
-    kind: asOptionalString(notification.kind),
-    sessionId: asOptionalString(notification.sessionId),
-    directory: asOptionalString(notification.directory) ?? (fallbackDirectory && fallbackDirectory !== "global" ? fallbackDirectory : undefined),
+    kind,
+    sessionId,
+    directory: directory || undefined,
     requireHidden: notification.requireHidden === true,
   }).catch((error) => {
     console.warn("[notifications] failed to dispatch UI notification", error)
@@ -632,15 +659,24 @@ async function resyncDirectorySessionStatuses(
   if (mode === "authoritative") {
     applyGlobalSessionStatusSnapshot(directory, nextStatuses, candidateSessionIds)
     // An authoritative snapshot that settles sessions previously observed
-    // busy/retry can orphan running tool parts (managed process died
-    // mid-turn, #2577): finalize them now. The snapshot write above already
-    // lowered their status to explicit idle, which is the gate the helper
-    // requires — a session the snapshot reports busy stays untouched.
+    // busy/retry can leave their trailing assistant message and tool parts
+    // unfinished (managed process died mid-turn, #2577): finalize them now.
+    // The snapshot write above already lowered their status to explicit idle,
+    // which is the gate the helper requires — a session the snapshot reports
+    // busy stays untouched.
     for (const sessionId of candidateSessionIds) {
       const interrupted = interruptedTurnToolParts(store.getState(), sessionId)
       if (interrupted) {
+        if (!interrupted.parts) {
+          store.setState((state) => ({
+            message: { ...state.message, [sessionId]: interrupted.messages },
+          }))
+          continue
+        }
+        const interruptedParts = interrupted.parts
         store.setState((state) => ({
-          part: { ...state.part, [interrupted.messageID]: interrupted.parts },
+          message: { ...state.message, [sessionId]: interrupted.messages },
+          part: { ...state.part, [interrupted.messageID]: interruptedParts },
         }))
       }
     }
@@ -1804,18 +1840,32 @@ export function handleEvent(
         messageID,
       })
     }
-    // The reducer already wrote the idle/error status into `draft`; mark the
-    // orphaned tools using the batched state and publish through the batch.
+    // The reducer already wrote the idle/error status into `draft`; finalize
+    // the interrupted message and orphaned tools through the same batch.
     if (sessionID) {
       const interrupted = interruptedTurnToolParts(state, sessionID)
       if (interrupted) {
-        cloneField("part", (value) => ({ ...(value ?? {}) }))
-        ;(draft as DirectoryStore).part[interrupted.messageID] = interrupted.parts
+        cloneField("message", (value) => ({ ...value }))
+        draft.message[sessionID] = interrupted.messages
+        if (interrupted.parts) {
+          cloneField("part", (value) => ({ ...(value ?? {}) }))
+          draft.part[interrupted.messageID] = interrupted.parts
+        }
         if (batch) {
           batch.states.set(store, draft as DirectoryStore)
           batch.changedStores.add(store)
         } else {
-          store.setState({ part: { ...(store.getState().part), [interrupted.messageID]: interrupted.parts } })
+          const currentState = store.getState()
+          if (interrupted.parts) {
+            store.setState({
+              message: { ...currentState.message, [sessionID]: interrupted.messages },
+              part: { ...currentState.part, [interrupted.messageID]: interrupted.parts },
+            })
+          } else {
+            store.setState({
+              message: { ...currentState.message, [sessionID]: interrupted.messages },
+            })
+          }
         }
       }
     }
@@ -1829,29 +1879,31 @@ export function handleEvent(
 //
 // A managed OpenCode process can die mid-turn (crash, health-check restart).
 // The persisted turn then never settles: the trailing assistant message has
-// no `time.completed` and its tool parts stay `pending`/`running` forever —
-// the server never finalizes them (anomalyco/opencode#19023). The
+// no `time.completed`, and any tool parts can stay `pending`/`running`
+// forever — the server never finalizes them (anomalyco/opencode#19023). The
 // settle-triggered tail refresh above refetches the same stale records, so
-// the UI would keep running tool timers and "working" styling indefinitely
-// (#2577).
+// the UI would keep the assistant message unfinished and any tool timers and
+// "working" styling active indefinitely (#2577).
 //
 // OpenCode keeps a turn's session busy while it is genuinely alive —
 // including while waiting for a question/permission reply — so once a
 // session is AUTHORITATIVELY settled (a `session.idle`/`session.error`
 // event, or an authoritative status snapshot that lowers a previously busy
 // session) and the trailing assistant message is still unfinished with
-// active tool parts and no pending question/permission, the turn is
-// definitively interrupted. Finalize the orphaned parts locally as
-// `error`/`Interrupted` with an end time — the same shape OpenCode itself
-// writes for cancelled tools. A later terminal part event or a refresh that
-// carries the true terminal state supersedes the mark; a stale refresh that
-// still reports `running` is rejected by the reducer's and the materializer's
-// final-status preservation.
+// no pending question/permission, the turn is definitively interrupted.
+// Complete the assistant message locally with MessageAbortedError and finalize
+// any orphaned parts as `error`/`Interrupted` with an end time — the same shape
+// OpenCode itself writes for cancelled tools. A later terminal event can
+// supersede the mark; a stale refresh cannot regress the locally final state.
+type AssistantMessage = Extract<Message, { role: "assistant" }>
+type SdkMessageAbortedError = Extract<NonNullable<AssistantMessage["error"]>, { name: "MessageAbortedError" }>
+type LocalMessageAbortedError = SdkMessageAbortedError & { message: string }
+
 export function interruptedTurnToolParts(
   state: DirectoryStore,
   sessionID: string,
   now = Date.now(),
-): { messageID: string; parts: Part[] } | null {
+): { messageID: string; messages: Message[]; parts?: Part[] } | null {
   if ((state.question?.[sessionID] ?? []).length > 0) return null
   if ((state.permission?.[sessionID] ?? []).length > 0) return null
 
@@ -1863,37 +1915,62 @@ export function interruptedTurnToolParts(
     return null
   }
 
-  const messageID = getStaleRunningToolMessageID(state, sessionID)
-  if (!messageID) return null
-  const message = (state.message[sessionID] ?? []).find((candidate) => candidate.id === messageID)
-  if (!message) return null
-  if (typeof (message as { time?: { completed?: unknown } }).time?.completed === "number") {
+  const messages = state.message[sessionID] ?? []
+  let messageIndex = -1
+  for (let index = messages.length - 1; index >= 0; index -= 1) {
+    const candidate = messages[index]
+    if (candidate.role === "user") return null
+    if (candidate.role !== "assistant") continue
+    messageIndex = index
+    break
+  }
+  if (messageIndex < 0) return null
+
+  const message = messages[messageIndex]
+  if (message.role !== "assistant") return null
+  if (message.time.completed !== undefined) {
     // The turn finished; a missed terminal tool event is the tail refresh's
     // job, not an interruption.
     return null
   }
 
-  const current = state.part[messageID]
-  if (!current) return null
+  const messageID = message.id
+  const nextMessages = [...messages]
+  const error = {
+    name: "MessageAbortedError",
+    data: { message: "aborted" },
+    message: "aborted",
+  } satisfies LocalMessageAbortedError
+  nextMessages[messageIndex] = {
+    ...message,
+    time: { ...message.time, completed: now },
+    error,
+  }
 
-  let changed = false
-  const nextParts = current.map((part) => {
+  let partsChanged = false
+  const currentParts = state.part[messageID]
+  const nextParts = currentParts?.map((part) => {
     if (part.type !== "tool") return part
-    const partState = (part as { state?: { status?: unknown; time?: { start?: number } } }).state
-    if (!partState) return part
-    if (partState.status !== "pending" && partState.status !== "running") return part
-    changed = true
+    if (part.state.status !== "pending" && part.state.status !== "running") return part
+    partsChanged = true
+    const partTime = "time" in part.state ? part.state.time : undefined
+    const start = typeof partTime?.start === "number" ? partTime.start : now
     return {
       ...part,
       state: {
-        ...partState,
-        status: "error",
+        ...part.state,
+        status: "error" as const,
         error: "Interrupted",
-        time: { ...(partState.time ?? {}), end: now },
+        time: { start, end: now },
       },
-    } as Part
+    }
   })
-  return changed ? { messageID, parts: nextParts } : null
+
+  return {
+    messageID,
+    messages: nextMessages,
+    parts: partsChanged ? nextParts : undefined,
+  }
 }
 
 // ---------------------------------------------------------------------------