fix(sync): finalize tool parts orphaned by an interrupted turn after settlement

When a managed OpenCode process dies mid-turn (crash, health-check
restart), the persisted turn never settles: the trailing assistant
message has no time.completed and its tool parts stay pending/running
forever — the server never finalizes them (anomalyco/opencode#19023).
The existing settle-triggered tail refresh refetches the same stale
records, so the UI kept running tool timers and working styling
indefinitely (#2577).

Now, when a session is authoritatively settled (session.idle/
session.error event, or an authoritative status snapshot lowering a
previously busy session) and the trailing assistant message is still
unfinished with active tool parts and no pending question/permission,
the orphaned parts are finalized locally as error/"Interrupted" with
an end time — the same shape OpenCode itself writes for cancelled
tools. The mark is gated on an explicit idle status (absent status is
"unknown", never judged), never applies while busy (including
question/permission waits), and a later terminal event or refresh
supersedes it while a stale running refresh cannot regress it (the
reducer and materializer already preserve final statuses).

Fixes #2577
This commit is contained in:
Serhii Dziupin
2026-08-05 14:15:03 +03:00
parent 34c221b07f
commit e3202df037
4 changed files with 287 additions and 0 deletions
+2
View File
@@ -204,6 +204,8 @@ Incomplete-session materialization is deduplicated by runtime, directory, and se
When `session.idle` or `session.error` settles a session but the trailing assistant message still contains a `pending` or `running` tool, sync refreshes that session tail. This narrowly reconciles a missed terminal tool-part event without refetching normally completed turns or stale tools from older turns. A stale refresh or delayed part event cannot regress a locally observed terminal tool to an active status. When `session.idle` or `session.error` settles a session but the trailing assistant message still contains a `pending` or `running` tool, sync refreshes that session tail. This narrowly reconciles a missed terminal tool-part event without refetching normally completed turns or stale tools from older turns. A stale refresh or delayed part event cannot regress a locally observed terminal tool to an active status.
When a session is authoritatively settled — `session.idle`/`session.error` event, or an authoritative status snapshot that lowers a previously busy session — and the trailing assistant message is still *unfinished* (`time.completed` missing) with active tool parts and no pending question/permission, the turn is treated as interrupted (managed OpenCode process died mid-turn; the server never finalizes the parts, see openchamber#2577 / anomalyco/opencode#19023). The active parts are finalized locally as `error`/`Interrupted` with an end time, so tool timers stop and cards render the error state. The mark is gated on an explicit idle status (absent status is "unknown", never judged), never applies while the session is busy (including question/permission waits), and a later terminal event or refresh supersedes it while a stale `running` refresh cannot regress it.
Directory stores also own session-keyed sidecar notification channels for permissions and message materialization. High-frequency realtime part events annotate the exact session/message before committing, so visible records, user history, renderability, and sidebar permission rows are not notified by unrelated sessions. Structural message replacements notify only changed subscribed session buckets; unannotated bulk part replacement conservatively resets active message subscribers so bootstrap, pagination, rollback, and legacy writers cannot leave stale projections. Directory stores also own session-keyed sidecar notification channels for permissions and message materialization. High-frequency realtime part events annotate the exact session/message before committing, so visible records, user history, renderability, and sidebar permission rows are not notified by unrelated sessions. Structural message replacements notify only changed subscribed session buckets; unannotated bulk part replacement conservatively resets active message subscribers so bootstrap, pagination, rollback, and legacy writers cannot leave stale projections.
Message sidecar consumers also filter targeted updates by purpose before notifying React. Suspended live-tail text/reasoning changes do not rebuild visible message records, but structural Task session identity changes bypass suspension so a parent can link a newly created subagent immediately. Assistant-only part changes do not rebuild user input history, and targeted updates that preserve authoritative part buckets do not recheck a session that is already renderable. Message replacements, removed final part buckets, and conservative resets always notify. Message sidecar consumers also filter targeted updates by purpose before notifying React. Suspended live-tail text/reasoning changes do not rebuild visible message records, but structural Task session identity changes bypass suspension so a parent can link a newly created subagent immediately. Assistant-only part changes do not rebuild user input history, and targeted updates that preserve authoritative part buckets do not recheck a session that is already renderable. Message replacements, removed final part buckets, and conservative resets always notify.
@@ -0,0 +1,152 @@
/**
* Tests for interrupted-turn reconciliation (#2577): when a managed OpenCode
* process dies mid-turn, the persisted turn never settles the trailing
* assistant message has no time.completed and its tool parts stay running.
* Once the session is authoritatively settled, `interruptedTurnToolParts`
* finalizes the orphaned parts locally.
*/
import { describe, expect, test } from "bun:test"
import type { Message, Part } from "@opencode-ai/sdk/v2/client"
import { interruptedTurnToolParts } from "../sync-context"
import type { DirectoryStore } from "../child-store"
import { INITIAL_STATE } from "../types"
function state(overrides: Partial<DirectoryStore> = {}): DirectoryStore {
return {
...INITIAL_STATE,
session_status: {},
message: {},
part: {},
question: {},
permission: {},
...overrides,
} as unknown as DirectoryStore
}
function runningTool(id: string, messageID: string, start = 1000): Part {
return {
id,
messageID,
sessionID: "ses_1",
type: "tool",
tool: "bash",
state: { status: "running", time: { start }, input: {} },
} as unknown as Part
}
function completedTool(id: string, messageID: string): Part {
return {
id,
messageID,
sessionID: "ses_1",
type: "tool",
tool: "bash",
state: { status: "completed", time: { start: 1000, end: 2000 }, input: {} },
} as unknown as Part
}
function pendingTool(id: string, messageID: string): Part {
return {
id,
messageID,
sessionID: "ses_1",
type: "tool",
tool: "bash",
state: { status: "pending", time: { start: 1000 }, input: {} },
} as unknown as Part
}
function unfinishedAssistantMessage(id: string): Message {
return { id, sessionID: "ses_1", role: "assistant", parentID: "", modelID: "", providerID: "", mode: "primary", system: "", agent: "", model: "", time: { created: 10 } } as unknown as Message
}
function finishedAssistantMessage(id: string): Message {
return { id, sessionID: "ses_1", role: "assistant", parentID: "", modelID: "", providerID: "", mode: "primary", system: "", agent: "", model: "", time: { created: 10, completed: 2000 } } as unknown as Message
}
describe("interruptedTurnToolParts (#2577)", () => {
test("settled session with unfinished message and running tool finalizes the part", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
})
const result = interruptedTurnToolParts(store, "ses_1", 5000)
expect(result).not.toBeNull()
const part = result!.parts[0] as { state: { status: string; error: string; time: { end: number } } }
expect(part.state.status).toBe("error")
expect(part.state.error).toBe("Interrupted")
expect(part.state.time.end).toBe(5000)
})
test("busy session is never marked (live work)", () => {
const store = state({
session_status: { ses_1: { type: "busy" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
test("absent status is unknown, not settled — never marked", () => {
const store = state({
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
test("finished message is not an interruption (tail refresh reconciles it)", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [finishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
test("pending question means the turn is waiting for input, not interrupted", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
question: { ses_1: [{ id: "q_1", sessionID: "ses_1", questions: [{ question: "?", header: "h", options: [{ label: "a", description: "" }] }] }] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
test("pending permission means the turn is waiting for input, not interrupted", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [runningTool("tool_1", "msg_1")] },
permission: { ses_1: [{ id: "p_1", sessionID: "ses_1", permission: "bash", patterns: [], metadata: {}, always: [] }] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
test("only active parts are finalized; completed parts are untouched", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: {
msg_1: [runningTool("tool_1", "msg_1"), completedTool("tool_2", "msg_1"), pendingTool("tool_3", "msg_1")],
},
})
const result = interruptedTurnToolParts(store, "ses_1", 5000)
expect(result).not.toBeNull()
const statuses = result!.parts.map((part) => (part as { state: { status: string } }).state.status)
expect(statuses).toEqual(["error", "completed", "error"])
})
test("no active parts → no change", () => {
const store = state({
session_status: { ses_1: { type: "idle" } },
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
part: { msg_1: [completedTool("tool_2", "msg_1")] },
})
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
})
})
@@ -167,6 +167,39 @@ describe("materializeSessionSnapshots", () => {
expect(mergedPart.state?.time?.end).toBe(2000) expect(mergedPart.state?.time?.end).toBe(2000)
}) })
test("does not regress a locally interrupted tool (error + end) when a stale running snapshot arrives", () => {
// The #2577 mark writes status "error" + end time; a later stale refresh
// that still reports the part as running must not undo it.
const interruptedTool = {
id: "prt_1",
messageID: "msg_1",
sessionID: "ses_1",
type: "tool",
state: { status: "error", error: "Interrupted", time: { start: 1000, end: 5000 } },
} as unknown as Part
const staleRunningTool = {
id: "prt_1",
messageID: "msg_1",
sessionID: "ses_1",
type: "tool",
state: { status: "running", time: { start: 1000 } },
} as unknown as Part
const state = {
message: { ses_1: [message("msg_1")] },
part: { msg_1: [interruptedTool] },
}
const result = materializeSessionSnapshots(
state,
"ses_1",
[{ info: message("msg_1"), parts: [staleRunningTool] }],
)
expect(result.part.msg_1[0]).toBe(interruptedTool)
expect(result.part.msg_1[0]).not.toBe(staleRunningTool)
expect((result.part.msg_1[0] as { state: { status: string } }).state.status).toBe("error")
})
test("does not regress a completed tool when a stale running snapshot arrives", () => { test("does not regress a completed tool when a stale running snapshot arrives", () => {
const completedTool = { const completedTool = {
id: "prt_1", id: "prt_1",
+100
View File
@@ -628,6 +628,19 @@ async function resyncDirectorySessionStatuses(
applySessionStatusSnapshot(store, nextStatuses, candidateSessionIds, mode) applySessionStatusSnapshot(store, nextStatuses, candidateSessionIds, mode)
if (mode === "authoritative") { if (mode === "authoritative") {
applyGlobalSessionStatusSnapshot(directory, nextStatuses, candidateSessionIds) applyGlobalSessionStatusSnapshot(directory, nextStatuses, candidateSessionIds)
// An authoritative snapshot that settles sessions previously observed
// busy/retry can orphan running tool parts (managed process died
// mid-turn, #2577): finalize them now. The snapshot write above already
// lowered their status to explicit idle, which is the gate the helper
// requires — a session the snapshot reports busy stays untouched.
for (const sessionId of candidateSessionIds) {
const interrupted = interruptedTurnToolParts(store.getState(), sessionId)
if (interrupted) {
store.setState((state) => ({
part: { ...state.part, [interrupted.messageID]: interrupted.parts },
}))
}
}
} }
return nextStatuses return nextStatuses
} }
@@ -1758,11 +1771,98 @@ function handleEvent(
messageID, messageID,
}) })
} }
// The reducer already wrote the idle/error status into `draft`; mark the
// orphaned tools using the batched state and publish through the batch.
if (sessionID) {
const interrupted = interruptedTurnToolParts(state, sessionID)
if (interrupted) {
cloneField("part", (value) => ({ ...(value ?? {}) }))
;(draft as DirectoryStore).part[interrupted.messageID] = interrupted.parts
if (batch) {
batch.states.set(store, draft as DirectoryStore)
batch.changedStores.add(store)
} else {
store.setState({ part: { ...(store.getState().part), [interrupted.messageID]: interrupted.parts } })
}
}
}
} }
updateRoutingIndexFromEvent(routingIndex, resolvedDirectory, payload) updateRoutingIndexFromEvent(routingIndex, resolvedDirectory, payload)
} }
// ---------------------------------------------------------------------------
// Interrupted-turn reconciliation
//
// A managed OpenCode process can die mid-turn (crash, health-check restart).
// The persisted turn then never settles: the trailing assistant message has
// no `time.completed` and its tool parts stay `pending`/`running` forever —
// the server never finalizes them (anomalyco/opencode#19023). The
// settle-triggered tail refresh above refetches the same stale records, so
// the UI would keep running tool timers and "working" styling indefinitely
// (#2577).
//
// OpenCode keeps a turn's session busy while it is genuinely alive —
// including while waiting for a question/permission reply — so once a
// session is AUTHORITATIVELY settled (a `session.idle`/`session.error`
// event, or an authoritative status snapshot that lowers a previously busy
// session) and the trailing assistant message is still unfinished with
// active tool parts and no pending question/permission, the turn is
// definitively interrupted. Finalize the orphaned parts locally as
// `error`/`Interrupted` with an end time — the same shape OpenCode itself
// writes for cancelled tools. A later terminal part event or a refresh that
// carries the true terminal state supersedes the mark; a stale refresh that
// still reports `running` is rejected by the reducer's and the materializer's
// final-status preservation.
export function interruptedTurnToolParts(
state: DirectoryStore,
sessionID: string,
now = Date.now(),
): { messageID: string; parts: Part[] } | null {
if ((state.question?.[sessionID] ?? []).length > 0) return null
if ((state.permission?.[sessionID] ?? []).length > 0) return null
const status = state.session_status?.[sessionID]
if (!status || status.type !== "idle") {
// Absent status is "unknown", not settled (the reducer maps both
// session.idle and session.error to {type:"idle"}): never judge an
// interrupted turn without an authoritative settle signal.
return null
}
const messageID = getStaleRunningToolMessageID(state, sessionID)
if (!messageID) return null
const message = (state.message[sessionID] ?? []).find((candidate) => candidate.id === messageID)
if (!message) return null
if (typeof (message as { time?: { completed?: unknown } }).time?.completed === "number") {
// The turn finished; a missed terminal tool event is the tail refresh's
// job, not an interruption.
return null
}
const current = state.part[messageID]
if (!current) return null
let changed = false
const nextParts = current.map((part) => {
if (part.type !== "tool") return part
const partState = (part as { state?: { status?: unknown; time?: { start?: number } } }).state
if (!partState) return part
if (partState.status !== "pending" && partState.status !== "running") return part
changed = true
return {
...part,
state: {
...partState,
status: "error",
error: "Interrupted",
time: { ...(partState.time ?? {}), end: now },
},
} as Part
})
return changed ? { messageID, parts: nextParts } : null
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Provider // Provider
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------