feat(work-status): add opt-in turn statistics (#3177)
Add optional completed-turn statistics without changing the existing panel layout. Separate final text delivery speed from whole-turn throughput, preserve scope and opt-in settings, and explain each metric with localized delayed tooltips. Validated focused telemetry, lifecycle, sync and persistence tests, all-workspace type-check and lint, web builds, the 12-locale narrow layout, and full GitHub CI.
This commit is contained in:
@@ -17,7 +17,8 @@ conditionally; passing "am I first?" down would mean each one tracking what the
|
||||
sections above it decided to render.
|
||||
|
||||
Sections render nothing when they have no rows, so the panel collapses upward
|
||||
instead of reserving empty space.
|
||||
instead of reserving empty space. Opt-in Turn stats keeps its header for a
|
||||
selected session even without metrics, so a saved collapsed state can reopen.
|
||||
|
||||
## What it is not
|
||||
|
||||
@@ -103,11 +104,49 @@ which requests only providers enabled for this panel.
|
||||
| Subagent blockers | directory `permission` / `question` maps | one subscription covers every child |
|
||||
| Usage | `components/usage/usageGroups.ts` over `useQuotaStore` | grouping shared with the mobile popover; presentation is not |
|
||||
| Linked threads | `lib/linkedIssues.ts` over session metadata | written by the flows that attach an issue or PR |
|
||||
| Turn stats | `telemetry.ts` over `useSessionMessageRecords` | opt-in; computed only while expanded and authoritatively idle |
|
||||
| Goal | `useSessionGoal` | respects the Settings toggle |
|
||||
| MCP | `useMcpStore` | connect/disconnect reuses the dropdown's actions |
|
||||
| Pinned messages | `getContextObligatoryMessages` + `state.part` | see below |
|
||||
| Todos | live `state.todo[sessionId]`, persisted fallback | live channel wins |
|
||||
|
||||
### Turn stats
|
||||
|
||||
The section follows Usage and reuses the panel's existing rows. Only its header
|
||||
has an icon; metric rows use labels and values without leading icons. It
|
||||
reads already-loaded records without fetching history. The newest turn needs a
|
||||
preceding user message and completed assistant steps. A truncated or unfinished
|
||||
turn has no whole-turn result; later materialization can supply it.
|
||||
|
||||
Two rates answer different questions. Response speed uses the final assistant
|
||||
message's output tokens divided by the union of its nonempty text intervals.
|
||||
It excludes initial waiting, reasoning tokens and reasoning time, and earlier
|
||||
tool steps. It requires complete, valid text timing and a final message without
|
||||
tools, errors, synthetic text or ignored text. This measures text delivery from
|
||||
stored timestamps, not provider-side decode speed. The header shows only this
|
||||
rate; unavailable response timing never falls back to whole-turn speed.
|
||||
|
||||
Whole-turn speed uses output plus reasoning tokens from every step, divided by
|
||||
elapsed assistant time minus the union of completed and failed tool intervals.
|
||||
Waiting for each model response remains included. Invalid or missing inputs
|
||||
omit the dependent metric rather than becoming zero; reported zeros remain
|
||||
valid. TTFT averages the earliest text/reasoning start delay from every step,
|
||||
only when all steps have a valid sample.
|
||||
|
||||
Metric labels stay short. Every row is a single hover and keyboard-focus target
|
||||
for a shared tooltip, with a 750ms hover delay and a portal outside the panel's
|
||||
scroller. Tooltips explain the measurement in every locale. The token row uses
|
||||
compact input/output arrows; its tooltip gives full counts and explains that
|
||||
input excludes cached tokens and output includes reasoning across all steps.
|
||||
|
||||
Records subscriptions and aggregation stop while collapsed, busy, retrying, or
|
||||
awaiting status authority. Explicit idle events or a successful directory status
|
||||
snapshot allow computation. One component-owned committed result keeps the
|
||||
headline and rows stable during the next active turn. Its identity includes
|
||||
runtime, normalized directory and session. Scope changes discard it; fresh empty
|
||||
or reverted records clear it. There is no global message-ID cache. Corrections
|
||||
to existing message/part identities invalidate the current result.
|
||||
|
||||
### Context usage has its own computation, on purpose
|
||||
|
||||
`useSessionUIStore.getContextUsage` cannot serve this panel for two reasons:
|
||||
@@ -191,8 +230,9 @@ the row reflects the reset tree rather than a mid-creation snapshot.
|
||||
Ordering is by durability, not category:
|
||||
|
||||
1. **Session** (goal, context, cost), **Project** (attention, branch,
|
||||
changes, PR, checks) and **Usage** — true for as long as the session is
|
||||
open. Usage sits here rather than lower down because a spent quota stops the
|
||||
changes, PR, checks), **Usage**, and **Turn stats** (opt-in session telemetry:
|
||||
throughput, duration, TTFT, cache hit rate) — true for as long as the session
|
||||
is open. Usage sits here rather than lower down because a spent quota stops the
|
||||
work outright;
|
||||
2. **Subagents**, **Tasks** — what is happening right now;
|
||||
3. **MCP**, **Pinned messages**, **Context sources** — supporting material.
|
||||
@@ -201,10 +241,13 @@ Ordering is by durability, not category:
|
||||
|
||||
A persisted preference (`workStatusPanelEnabled`) drives a header toggle, and a
|
||||
dialog behind the equalizer icon switches individual sections off. Hidden
|
||||
sections are stored rather than visible ones, so a section added later appears
|
||||
for everyone instead of staying invisible to whoever had saved settings before
|
||||
it existed. Both travel the full settings pipeline, including the server
|
||||
whitelist without which the keys never reach `settings.json`.
|
||||
sections are stored rather than visible ones. Telemetry is the opt-in exception:
|
||||
UI-store v20 and legacy server-list hydration add it to the hidden set. A
|
||||
`workStatusHiddenSectionsExplicit` marker records that a list was chosen in a
|
||||
client with telemetry support. The marker and list travel together through
|
||||
autosave, sanitization, and server settings, so an explicit empty list enables
|
||||
everything while an old empty list does not enable telemetry. Complete settings
|
||||
snapshots own this preference; unrelated partial save echoes leave it unchanged.
|
||||
|
||||
`workStatusPanelVisible` is separate and transient: the switch can be on while
|
||||
layout still refuses the panel. The header and the git rail read it to drop the
|
||||
|
||||
@@ -8,6 +8,7 @@ import { WORK_STATUS_PANEL_WIDTH } from './useWorkStatusVisibility';
|
||||
import { WorkStatusGoalRow } from './WorkStatusGoalRow';
|
||||
import { WorkStatusPrimaryGroup } from './WorkStatusPrimaryGroup';
|
||||
import { WorkStatusUsageSection } from './WorkStatusUsageSection';
|
||||
import { WorkStatusTelemetrySection } from './WorkStatusTelemetrySection';
|
||||
import { WorkStatusSubagentsSection } from './WorkStatusSubagentsSection';
|
||||
import { WorkStatusTasksSection } from './WorkStatusTasksSection';
|
||||
import { WorkStatusMcpSection } from './WorkStatusMcpSection';
|
||||
@@ -254,6 +255,7 @@ export const WorkStatusPanel: React.FC<Props> = ({ sessionId, directory, visible
|
||||
goalRow={<WorkStatusGoalRow sessionId={sessionId} directory={directory} />}
|
||||
/>
|
||||
{sectionVisible('usage') ? <WorkStatusUsageSection /> : null}
|
||||
{sectionVisible('telemetry') ? <WorkStatusTelemetrySection sessionId={sessionId} directory={directory} /> : null}
|
||||
{sectionVisible('subagents') ? <WorkStatusSubagentsSection sessionId={sessionId} directory={directory} /> : null}
|
||||
{sectionVisible('tasks') ? <WorkStatusTasksSection sessionId={sessionId} directory={directory} /> : null}
|
||||
{sectionVisible('mcp') ? <WorkStatusMcpSection directory={directory} /> : null}
|
||||
|
||||
@@ -20,9 +20,8 @@ import {
|
||||
/**
|
||||
* Which sections the work-status panel may show.
|
||||
*
|
||||
* Everything is on by default and the choice is stored as the *hidden* set, so
|
||||
* a section added in a later release appears for everyone rather than staying
|
||||
* invisible to whoever had saved settings before it existed.
|
||||
* Choices are stored as the hidden set. Telemetry is opt-in; Show all is an
|
||||
* explicit choice to enable it along with the other sections.
|
||||
*/
|
||||
export const WorkStatusSectionsDialog: React.FC<{
|
||||
open: boolean;
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import React, { act } from 'react';
|
||||
import { createRoot, type Root } from 'react-dom/client';
|
||||
import { Window } from 'happy-dom';
|
||||
import { createOpencodeClient, type AssistantMessage, type Session, type UserMessage } from '@opencode-ai/sdk/v2';
|
||||
import { useUIStore } from '@/stores/useUIStore';
|
||||
import { I18nProvider } from '@/lib/i18n';
|
||||
import { SyncProvider } from '@/sync/sync-context';
|
||||
import { getSyncChildStores } from '@/sync/sync-refs';
|
||||
import { getSyncPerformanceDiagnostics, resetSyncPerformanceDiagnostics, setSyncPerformanceDiagnosticsEnabled } from '@/sync/performance-diagnostics';
|
||||
let WorkStatusTelemetrySection: typeof import('./WorkStatusTelemetrySection').WorkStatusTelemetrySection;
|
||||
|
||||
const directory = '/repo';
|
||||
const sessionId = 'session-1';
|
||||
const user: UserMessage = { id: 'user-1', sessionID: sessionId, role: 'user', time: { created: 1000 }, agent: 'build', model: { providerID: 'test', modelID: 'test' } };
|
||||
const session: Session = { id: sessionId, slug: 'test', projectID: 'project', directory, title: 'test', version: '1', time: { created: 0, updated: 1 } };
|
||||
let tokenReads = 0;
|
||||
const assistant: AssistantMessage = {
|
||||
id: 'assistant-final', sessionID: sessionId, role: 'assistant', parentID: user.id,
|
||||
agent: 'build', mode: 'build', providerID: 'test', modelID: 'test', path: { cwd: directory, root: directory },
|
||||
time: { created: 2000, completed: 7000 }, cost: 0.01,
|
||||
get tokens() { tokenReads += 1; return { input: 100, output: 20, reasoning: 10, cache: { read: 40, write: 0 } }; },
|
||||
};
|
||||
|
||||
const DOM_GLOBAL_NAMES = ['window', 'document', 'navigator', 'Node', 'Element', 'HTMLElement', 'HTMLIFrameElement', 'localStorage', 'getComputedStyle', 'ResizeObserver', 'requestAnimationFrame', 'cancelAnimationFrame', 'IS_REACT_ACT_ENVIRONMENT'] as const;
|
||||
const installDom = () => {
|
||||
const win = new Window({ url: 'http://localhost' });
|
||||
const previous = DOM_GLOBAL_NAMES.map((name) => [name, Object.getOwnPropertyDescriptor(globalThis, name)] as const);
|
||||
const values = { window: win, document: win.document, navigator: win.navigator, Node: win.Node, Element: win.Element,
|
||||
HTMLElement: win.HTMLElement, HTMLIFrameElement: win.HTMLIFrameElement, localStorage: win.localStorage,
|
||||
getComputedStyle: win.getComputedStyle.bind(win), ResizeObserver: win.ResizeObserver,
|
||||
requestAnimationFrame: win.requestAnimationFrame.bind(win), cancelAnimationFrame: win.cancelAnimationFrame.bind(win), IS_REACT_ACT_ENVIRONMENT: true };
|
||||
for (const name of DOM_GLOBAL_NAMES) Object.defineProperty(globalThis, name, { value: values[name], configurable: true, writable: true });
|
||||
const container = document.createElement('div');
|
||||
document.body.appendChild(container);
|
||||
return { container, restore: () => {
|
||||
for (const [name, descriptor] of previous) {
|
||||
if (descriptor) Object.defineProperty(globalThis, name, descriptor);
|
||||
else Reflect.deleteProperty(globalThis, name);
|
||||
}
|
||||
void win.happyDOM.close();
|
||||
} };
|
||||
};
|
||||
|
||||
describe('mounted turn telemetry with live sync stores', () => {
|
||||
let root: Root;
|
||||
let dom: ReturnType<typeof installDom>;
|
||||
let messageRequests = 0;
|
||||
// Keep bootstrap pending so each test controls real store publications. No
|
||||
// hook/module replacements: subscription and materialization paths are real.
|
||||
const sdk = createOpencodeClient({ baseUrl: 'http://telemetry.test', fetch: (request) => {
|
||||
const url = new URL(request instanceof Request ? request.url : request.toString());
|
||||
if (/\/session\/[^/]+\/message$/.test(url.pathname)) messageRequests += 1;
|
||||
return new Promise<Response>(() => undefined);
|
||||
} });
|
||||
const render = async (visible = true, selectedDirectory = directory, selectedSession = sessionId) => {
|
||||
await act(async () => root.render(
|
||||
<SyncProvider sdk={sdk} directory={selectedDirectory}>
|
||||
<I18nProvider>{visible ? <WorkStatusTelemetrySection sessionId={selectedSession} directory={selectedDirectory} /> : null}</I18nProvider>
|
||||
</SyncProvider>,
|
||||
));
|
||||
};
|
||||
const store = (dir = directory) => {
|
||||
const result = getSyncChildStores().getChild(dir);
|
||||
if (!result) throw new Error('Expected mounted directory store');
|
||||
return result;
|
||||
};
|
||||
|
||||
beforeEach(async () => {
|
||||
dom = installDom();
|
||||
({ WorkStatusTelemetrySection } = await import('./WorkStatusTelemetrySection'));
|
||||
root = createRoot(dom.container);
|
||||
tokenReads = 0;
|
||||
messageRequests = 0;
|
||||
setSyncPerformanceDiagnosticsEnabled(true);
|
||||
useUIStore.setState({ workStatusExpandedSections: {}, workStatusHiddenSections: ['telemetry'], workStatusHiddenSectionsExplicit: false });
|
||||
await render();
|
||||
await act(async () => store().setState({ session: [session], message: { [sessionId]: [user, assistant] }, part: { [assistant.id]: [] }, session_status: {} }));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await act(async () => root.unmount());
|
||||
setSyncPerformanceDiagnosticsEnabled(false);
|
||||
dom.restore();
|
||||
});
|
||||
|
||||
test('waits for authority, then shows actual token values even when idle is omitted from the snapshot', async () => {
|
||||
expect(dom.container.textContent).toContain('Turn stats');
|
||||
expect(dom.container.textContent).not.toContain('Whole turn');
|
||||
await act(async () => store().setState({ sessionStatusReady: true }));
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
expect(dom.container.textContent).toContain('100 ↑ · 30 ↓');
|
||||
const heading = dom.container.querySelector('button');
|
||||
if (!heading) throw new Error('Expected section heading');
|
||||
expect(heading.textContent).toBe('Turn stats');
|
||||
expect(heading.querySelectorAll('svg').length).toBe(2);
|
||||
expect(dom.container.querySelectorAll('svg').length).toBe(2);
|
||||
expect(tokenReads > 0).toBe(true);
|
||||
expect(messageRequests).toBe(0);
|
||||
});
|
||||
|
||||
test('collapsed remount keeps a usable header and reopening reads fresh data', async () => {
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'idle' } } }));
|
||||
const button = dom.container.querySelector('button');
|
||||
if (!button) throw new Error('Expected collapse button');
|
||||
await act(async () => button.click());
|
||||
await render(false);
|
||||
await render();
|
||||
expect(dom.container.querySelector('button')?.getAttribute('aria-expanded')).toBe('false');
|
||||
expect(dom.container.textContent).toContain('Turn stats');
|
||||
expect(dom.container.textContent).not.toContain('Whole turn');
|
||||
const reopen = dom.container.querySelector('button');
|
||||
if (!reopen) throw new Error('Expected reopen button');
|
||||
await act(async () => reopen.click());
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
});
|
||||
|
||||
test('busy, retry and collapsed updates do not notify records subscribers or aggregate tokens', async () => {
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'idle' } } }));
|
||||
// Positive control: on idle, part replacement reaches the subscriber and calculator.
|
||||
tokenReads = 0;
|
||||
resetSyncPerformanceDiagnostics();
|
||||
await act(async () => store().setState({ part: { [assistant.id]: [] } }));
|
||||
expect(tokenReads > 0).toBe(true);
|
||||
expect((getSyncPerformanceDiagnostics()?.sessionMessageChangeCallbacks ?? 0) > 0).toBe(true);
|
||||
|
||||
for (const mode of ['busy', 'retry', 'collapsed'] as const) {
|
||||
await act(async () => {
|
||||
store().setState({ session_status: { [sessionId]: mode === 'retry'
|
||||
? { type: 'retry', attempt: 1, message: 'retry', next: 0 }
|
||||
: { type: mode === 'busy' ? 'busy' : 'idle' } } });
|
||||
useUIStore.getState().setWorkStatusSectionExpanded('telemetry', mode !== 'collapsed');
|
||||
});
|
||||
resetSyncPerformanceDiagnostics();
|
||||
tokenReads = 0;
|
||||
for (let i = 0; i < 100; i += 1) {
|
||||
await act(async () => store().setState({ part: { [assistant.id]: [{ id: 'text', sessionID: sessionId,
|
||||
messageID: assistant.id, type: 'text', text: String(i), time: { start: 2500 } }] } }));
|
||||
}
|
||||
expect(tokenReads).toBe(0);
|
||||
expect(getSyncPerformanceDiagnostics()?.sessionMessageChangeCallbacks).toBe(0);
|
||||
if (mode === 'collapsed') expect(dom.container.textContent).toBe('Turn stats');
|
||||
else expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
}
|
||||
});
|
||||
|
||||
test('session and directory changes cannot retain another scope, including equal IDs', async () => {
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'idle' } } }));
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
await render(true, directory, 'another-session');
|
||||
expect(dom.container.textContent).not.toContain('~6 tok/s');
|
||||
await render(true, '/another-repo');
|
||||
await act(async () => store('/another-repo').setState({ session_status: { [sessionId]: { type: 'busy' } } }));
|
||||
expect(dom.container.textContent).not.toContain('~6 tok/s');
|
||||
await render(true, directory);
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
});
|
||||
|
||||
test('same-ID corrections, partial history and reverts replace rather than cache stale stats', async () => {
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'idle' } }, message: { [sessionId]: [assistant] } }));
|
||||
expect(dom.container.textContent).not.toContain('Whole turn');
|
||||
await act(async () => store().setState({ message: { [sessionId]: [user, assistant] } }));
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
const corrected = { ...assistant, tokens: { ...assistant.tokens, output: 90 } };
|
||||
await act(async () => store().setState({ message: { [sessionId]: [user, corrected] } }));
|
||||
expect(dom.container.textContent).toContain('~20 tok/s');
|
||||
await act(async () => store().setState({ session: [{ ...session, revert: { messageID: user.id } }] }));
|
||||
expect(dom.container.textContent).not.toContain('~20 tok/s');
|
||||
await act(async () => store().setState({ session: [session] }));
|
||||
expect(dom.container.textContent).toContain('~20 tok/s');
|
||||
await act(async () => store().setState({ message: {} }));
|
||||
expect(dom.container.textContent).not.toContain('~20 tok/s');
|
||||
});
|
||||
|
||||
test('runtime identity changes discard retained results even with equal directory and session IDs', async () => {
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'idle' } } }));
|
||||
await act(async () => store().setState({ session_status: { [sessionId]: { type: 'busy' } } }));
|
||||
expect(dom.container.textContent).toContain('~6 tok/s');
|
||||
Object.defineProperty(window, '__OPENCHAMBER_API_BASE_URL__', { value: 'https://second-runtime.test', configurable: true });
|
||||
await render();
|
||||
expect(dom.container.textContent).not.toContain('~6 tok/s');
|
||||
});
|
||||
|
||||
test('the heading shows response speed only, never whole-turn speed as a fallback', async () => {
|
||||
await act(async () => store().setState({ sessionStatusReady: true, part: { [assistant.id]: [
|
||||
{ id: 'text', type: 'text', sessionID: sessionId, messageID: assistant.id, text: 'Final reply', time: { start: 3000, end: 5000 } },
|
||||
] } }));
|
||||
expect(dom.container.querySelector('button')?.textContent).toBe('Turn stats~10 tok/s');
|
||||
expect(dom.container.textContent).toContain('Response~10 tok/s');
|
||||
expect(dom.container.textContent).toContain('Whole turn~6 tok/s');
|
||||
await act(async () => store().setState({ part: { [assistant.id]: [] } }));
|
||||
expect(dom.container.querySelector('button')?.textContent).toBe('Turn stats');
|
||||
expect(dom.container.textContent).not.toContain('Response');
|
||||
expect(dom.container.textContent).toContain('Whole turn~6 tok/s');
|
||||
});
|
||||
|
||||
test('every metric has a full-row focus target and hover waits 750ms', async () => {
|
||||
const earlier = { ...assistant, id: 'earlier', time: { created: 1100, completed: 1900 } };
|
||||
await act(async () => store().setState({ sessionStatusReady: true,
|
||||
message: { [sessionId]: [user, earlier, assistant] }, part: {
|
||||
earlier: [{ id: 'earlier-text', type: 'text', sessionID: sessionId, messageID: earlier.id, text: 'Earlier', time: { start: 1200, end: 1800 } }],
|
||||
[assistant.id]: [{ id: 'text', type: 'text', sessionID: sessionId, messageID: assistant.id, text: 'Final reply', time: { start: 3000, end: 5000 } }],
|
||||
},
|
||||
}));
|
||||
const triggers = dom.container.querySelectorAll<HTMLElement>('[data-slot="tooltip-trigger"]');
|
||||
expect(triggers.length).toBe(9);
|
||||
for (const trigger of triggers) expect(trigger.tabIndex).toBe(0);
|
||||
expect(dom.container.querySelectorAll('[title]').length).toBe(0);
|
||||
const response = triggers[0];
|
||||
await act(async () => {
|
||||
response.dispatchEvent(new window.PointerEvent('pointerover', { bubbles: true, pointerType: 'mouse' }));
|
||||
response.dispatchEvent(new window.MouseEvent('mouseover', { bubbles: true }));
|
||||
response.dispatchEvent(new window.MouseEvent('mouseenter', { bubbles: true }));
|
||||
response.dispatchEvent(new window.MouseEvent('mousemove', { bubbles: true }));
|
||||
await new Promise((resolve) => setTimeout(resolve, 650));
|
||||
});
|
||||
expect(document.querySelector('[data-slot="tooltip-content"]')).toBeNull();
|
||||
expect(response.hasAttribute('data-popup-open')).toBe(false);
|
||||
await act(async () => { await new Promise((resolve) => setTimeout(resolve, 150)); });
|
||||
expect(response.hasAttribute('data-popup-open')).toBe(true);
|
||||
expect(document.querySelector('[data-slot="tooltip-content"]')?.textContent).toContain('How fast the final text arrived');
|
||||
expect(dom.container.querySelector('[data-slot="tooltip-content"]')).toBeNull();
|
||||
});
|
||||
|
||||
test('keyboard focus exposes the cost explanation without adding an icon or native title', async () => {
|
||||
await act(async () => store().setState({ sessionStatusReady: true }));
|
||||
const triggers = dom.container.querySelectorAll<HTMLElement>('[data-slot="tooltip-trigger"]');
|
||||
const cost = triggers[triggers.length - 1];
|
||||
await act(async () => cost.focus());
|
||||
expect(document.querySelector('[data-slot="tooltip-content"]')?.textContent).toContain('Cost reported by the provider');
|
||||
expect(cost.querySelector('svg')).toBeNull();
|
||||
expect(cost.hasAttribute('title')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,185 @@
|
||||
import React from 'react';
|
||||
import { getCurrentIntlLocale, useI18n } from '@/lib/i18n';
|
||||
import { Tooltip, TooltipContent, TooltipTrigger } from '@/components/ui/tooltip';
|
||||
import { useDirectorySync, useSessionMessageRecords, useSyncDirectory, useSyncRuntime } from '@/sync/sync-context';
|
||||
import { normalizePath } from '@/lib/pathNormalization';
|
||||
import { useUIStore } from '@/stores/useUIStore';
|
||||
import {
|
||||
WorkStatusCollapsibleSection,
|
||||
WorkStatusRow,
|
||||
WorkStatusValue,
|
||||
} from './WorkStatusPrimitives';
|
||||
import { useReportWorkStatusPresence } from './presenceContext';
|
||||
import {
|
||||
formatTelemetryDuration,
|
||||
formatTelemetryTokens,
|
||||
formatThroughputRate,
|
||||
getLatestCompletedTurnStats,
|
||||
type CompletedTurnStats,
|
||||
} from './telemetry';
|
||||
|
||||
type Props = {
|
||||
sessionId: string | null;
|
||||
directory: string | null;
|
||||
};
|
||||
|
||||
/** One hover/focus target covers both the label and its value. */
|
||||
const TelemetryRow: React.FC<{ label: string; description: string; value: React.ReactNode }> = ({ label, description, value }) => (
|
||||
<Tooltip delayDuration={750}>
|
||||
<TooltipTrigger asChild>
|
||||
<div tabIndex={0} className="min-w-0 rounded-md outline-none focus-visible:ring-2 focus-visible:ring-ring">
|
||||
<WorkStatusRow label={label} value={value} />
|
||||
</div>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="left" sideOffset={8} className="max-w-[min(320px,calc(100vw-24px))] whitespace-normal break-words text-left">
|
||||
<p className="font-medium">{label}</p>
|
||||
<p>{description}</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
);
|
||||
|
||||
export const WorkStatusTelemetrySection: React.FC<Props> = ({ sessionId, directory }) => {
|
||||
const { t } = useI18n();
|
||||
const expanded = useUIStore(
|
||||
React.useCallback((state) => state.workStatusExpandedSections['telemetry'] ?? true, []),
|
||||
);
|
||||
const { runtimeKey } = useSyncRuntime();
|
||||
const syncDirectory = useSyncDirectory();
|
||||
const scope = JSON.stringify([runtimeKey, normalizePath(directory ?? syncDirectory), sessionId]);
|
||||
const status = useDirectorySync(
|
||||
React.useCallback((state) => sessionId
|
||||
? state.session_status[sessionId]?.type ?? (state.sessionStatusReady ? 'idle' : 'unknown')
|
||||
: 'unknown', [sessionId]),
|
||||
directory ?? undefined,
|
||||
);
|
||||
const eligibleForStats = Boolean(sessionId && expanded && status === 'idle');
|
||||
|
||||
const records = useSessionMessageRecords(
|
||||
sessionId ?? '',
|
||||
directory ?? undefined,
|
||||
{ enabled: eligibleForStats },
|
||||
);
|
||||
|
||||
const computed = React.useMemo(() => {
|
||||
if (!eligibleForStats) return null;
|
||||
return getLatestCompletedTurnStats(records);
|
||||
}, [eligibleForStats, records]);
|
||||
|
||||
// Retain only one committed result, never message history or a global ID cache.
|
||||
const [retained, setRetained] = React.useState<{ scope: string; stats: CompletedTurnStats | null } | null>(null);
|
||||
React.useEffect(() => {
|
||||
if (eligibleForStats) {
|
||||
setRetained({ scope, stats: computed });
|
||||
} else {
|
||||
setRetained((previous) => previous?.scope === scope && status !== 'unknown' ? previous : null);
|
||||
}
|
||||
}, [scope, status, eligibleForStats, computed]);
|
||||
const stats = eligibleForStats ? computed : status !== 'unknown' && retained?.scope === scope ? retained.stats : null;
|
||||
const summary = stats && stats.responseTokensPerSecond !== null
|
||||
? formatThroughputRate(stats.responseTokensPerSecond)
|
||||
: undefined;
|
||||
|
||||
useReportWorkStatusPresence('telemetry', Boolean(sessionId));
|
||||
|
||||
if (!sessionId) return null;
|
||||
|
||||
return (
|
||||
<WorkStatusCollapsibleSection
|
||||
id="telemetry"
|
||||
title={t('chat.workStatus.section.telemetry')}
|
||||
icon="bar-chart-2"
|
||||
summary={summary}
|
||||
defaultExpanded
|
||||
>
|
||||
{stats ? (
|
||||
<>
|
||||
{stats.responseTokensPerSecond !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.responseSpeed')}
|
||||
description={t('chat.workStatus.telemetry.responseSpeedDescription')}
|
||||
value={<WorkStatusValue>{formatThroughputRate(stats.responseTokensPerSecond)}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
{stats.tokensPerSecond !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.speed')}
|
||||
description={t('chat.workStatus.telemetry.speedDescription')}
|
||||
value={<WorkStatusValue>{formatThroughputRate(stats.tokensPerSecond)}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.totalLlmDurationMs !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.llmDuration')}
|
||||
description={t('chat.workStatus.telemetry.llmDurationDescription')}
|
||||
value={<WorkStatusValue>{formatTelemetryDuration(stats.totalLlmDurationMs)}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.totalToolDurationMs !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.toolDuration')}
|
||||
description={t('chat.workStatus.telemetry.toolDurationDescription')}
|
||||
value={<WorkStatusValue>{formatTelemetryDuration(stats.totalToolDurationMs)}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.avgTtftMs !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.ttft')}
|
||||
description={t('chat.workStatus.telemetry.ttftDescription')}
|
||||
value={<WorkStatusValue>{formatTelemetryDuration(stats.avgTtftMs)}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.stepsCount > 1 ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.steps')}
|
||||
description={t('chat.workStatus.telemetry.stepsDescription')}
|
||||
value={<WorkStatusValue>{stats.stepsCount}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.inputTokens !== null && stats.outputTokens !== null && stats.reasoningTokens !== null && stats.totalGeneratedTokens !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.tokens')}
|
||||
description={t('chat.workStatus.telemetry.tokensDescription', {
|
||||
input: stats.inputTokens.toLocaleString(getCurrentIntlLocale()),
|
||||
output: stats.outputTokens.toLocaleString(getCurrentIntlLocale()),
|
||||
reasoning: stats.reasoningTokens.toLocaleString(getCurrentIntlLocale()),
|
||||
})}
|
||||
value={(
|
||||
<WorkStatusValue>
|
||||
{t('chat.workStatus.telemetry.tokens.inOut', {
|
||||
input: formatTelemetryTokens(stats.inputTokens),
|
||||
output: formatTelemetryTokens(stats.totalGeneratedTokens),
|
||||
})}
|
||||
</WorkStatusValue>
|
||||
)}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.cacheHitPercent !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.cacheHit')}
|
||||
description={t('chat.workStatus.telemetry.cacheHitDescription')}
|
||||
value={(
|
||||
<WorkStatusValue tone={stats.cacheHitPercent >= 50 ? 'success' : 'default'}>
|
||||
{`${stats.cacheHitPercent}%`}
|
||||
</WorkStatusValue>
|
||||
)}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{stats.cost !== null ? (
|
||||
<TelemetryRow
|
||||
label={t('chat.workStatus.telemetry.cost')}
|
||||
description={t('chat.workStatus.telemetry.costDescription')}
|
||||
value={<WorkStatusValue tone="muted">{`$${stats.cost.toFixed(3).replace(/0+$/, '').replace(/\.$/, '')}`}</WorkStatusValue>}
|
||||
/>
|
||||
) : null}
|
||||
</>
|
||||
) : null}
|
||||
</WorkStatusCollapsibleSection>
|
||||
);
|
||||
};
|
||||
@@ -111,9 +111,9 @@ describe('sanitizeWorkStatusHiddenSections', () => {
|
||||
expect(sanitizeWorkStatusHiddenSections(['usage', 'usage'])).toEqual(['usage']);
|
||||
});
|
||||
|
||||
test('treats a non-array payload as no preference', () => {
|
||||
expect(sanitizeWorkStatusHiddenSections(undefined)).toEqual([]);
|
||||
expect(sanitizeWorkStatusHiddenSections('usage')).toEqual([]);
|
||||
expect(sanitizeWorkStatusHiddenSections({ usage: true })).toEqual([]);
|
||||
test('treats a non-array payload as default hidden preference', () => {
|
||||
expect(sanitizeWorkStatusHiddenSections(undefined)).toEqual(['telemetry']);
|
||||
expect(sanitizeWorkStatusHiddenSections('usage')).toEqual(['telemetry']);
|
||||
expect(sanitizeWorkStatusHiddenSections({ usage: true })).toEqual(['telemetry']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -14,6 +14,7 @@ export const WORK_STATUS_SECTION_IDS = [
|
||||
'session',
|
||||
'repository',
|
||||
'usage',
|
||||
'telemetry',
|
||||
'subagents',
|
||||
'tasks',
|
||||
'mcp',
|
||||
@@ -23,16 +24,17 @@ export const WORK_STATUS_SECTION_IDS = [
|
||||
|
||||
type WorkStatusSectionId = (typeof WORK_STATUS_SECTION_IDS)[number];
|
||||
|
||||
export const WORK_STATUS_SECTION_LABEL_KEYS: Record<WorkStatusSectionId, I18nKey> = {
|
||||
export const WORK_STATUS_SECTION_LABEL_KEYS = {
|
||||
session: 'chat.workStatus.section.session',
|
||||
repository: 'chat.workStatus.section.project',
|
||||
usage: 'chat.workStatus.section.usage',
|
||||
telemetry: 'chat.workStatus.section.telemetry',
|
||||
subagents: 'chat.workStatus.section.subagents',
|
||||
tasks: 'chat.workStatus.section.tasks',
|
||||
mcp: 'chat.workStatus.section.mcp',
|
||||
pinned: 'chat.workStatus.section.pinned',
|
||||
contextSources: 'chat.workStatus.section.contextBreakdown',
|
||||
};
|
||||
} as const satisfies Record<WorkStatusSectionId, I18nKey>;
|
||||
|
||||
const KNOWN_IDS = new Set<string>(WORK_STATUS_SECTION_IDS);
|
||||
|
||||
@@ -40,9 +42,8 @@ const isWorkStatusSectionId = (value: unknown): value is WorkStatusSectionId =>
|
||||
typeof value === 'string' && KNOWN_IDS.has(value);
|
||||
|
||||
/**
|
||||
* Hidden sections are stored, not visible ones: everything is on by default, so
|
||||
* an empty list means "the user has changed nothing" and a section added later
|
||||
* appears without touching anyone's saved settings.
|
||||
* Hidden sections are stored, not visible ones. Telemetry is opt-in; legacy
|
||||
* lists must be normalized before use so adding it does not enable it.
|
||||
*/
|
||||
export const isWorkStatusSectionVisible = (
|
||||
hidden: readonly string[] | null | undefined,
|
||||
@@ -75,11 +76,16 @@ export const getWorkStatusPanelPresentation = ({
|
||||
showEmptyState: contentMounted && allSectionsHidden,
|
||||
});
|
||||
|
||||
export const sanitizeWorkStatusHiddenSections = (value: unknown): WorkStatusSectionId[] => {
|
||||
if (!Array.isArray(value)) return [];
|
||||
const WORK_STATUS_DEFAULT_HIDDEN_SECTIONS = [
|
||||
'telemetry',
|
||||
] as const satisfies readonly WorkStatusSectionId[];
|
||||
|
||||
export const sanitizeWorkStatusHiddenSections = (value: unknown, explicit = true): WorkStatusSectionId[] => {
|
||||
if (!Array.isArray(value)) return [...WORK_STATUS_DEFAULT_HIDDEN_SECTIONS];
|
||||
const seen = new Set<WorkStatusSectionId>();
|
||||
for (const entry of value) {
|
||||
if (isWorkStatusSectionId(entry)) seen.add(entry);
|
||||
}
|
||||
if (!explicit) seen.add('telemetry');
|
||||
return [...seen];
|
||||
};
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import type { AssistantMessage, Part, TextPart, UserMessage } from '@opencode-ai/sdk/v2';
|
||||
import { formatTelemetryDuration, formatTelemetryTokens, formatThroughputRate, getLatestCompletedTurnStats, mergeTimeIntervals, sumIntervalsDuration } from './telemetry';
|
||||
|
||||
const user: UserMessage = { id: 'u1', sessionID: 'session-1', role: 'user', time: { created: 0 }, agent: 'build', model: { providerID: 'test', modelID: 'test' } };
|
||||
const assistant = (overrides: Partial<AssistantMessage> = {}): AssistantMessage => ({
|
||||
id: 'a1', sessionID: 'session-1', role: 'assistant', parentID: user.id,
|
||||
agent: 'build', mode: 'build', providerID: 'test', modelID: 'test', path: { cwd: '/repo', root: '/repo' },
|
||||
time: { created: 1000, completed: 5000 }, cost: 0,
|
||||
tokens: { input: 100, output: 100, reasoning: 0, cache: { read: 0, write: 0 } },
|
||||
...overrides,
|
||||
});
|
||||
const tool = (start: number, end: number): Part => ({
|
||||
id: `tool-${start}`, sessionID: user.sessionID, messageID: 'a1', type: 'tool', tool: 'bash', callID: 'call',
|
||||
state: { status: 'completed', input: {}, output: '', title: 'test', metadata: {}, time: { start, end } },
|
||||
});
|
||||
const text = (start: number): TextPart => ({ id: `text-${start}`, sessionID: user.sessionID, messageID: 'a1', type: 'text', text: '', time: { start } });
|
||||
const turn = (info = assistant(), parts: Part[] = []) => [{ info: user, parts: [] }, { info, parts }];
|
||||
|
||||
describe('turn telemetry', () => {
|
||||
test('merges unsorted parallel, nested, adjoining and invalid tool intervals', () => {
|
||||
expect(mergeTimeIntervals([])).toEqual([]);
|
||||
const merged = mergeTimeIntervals([[3000, 4000], [1000, 3000], [1500, 2500], [6000, 7000], [NaN, 1], [9, 8]]);
|
||||
expect(merged).toEqual([[1000, 4000], [6000, 7000]]);
|
||||
expect(sumIntervalsDuration(merged)).toBe(4000);
|
||||
});
|
||||
|
||||
test('formats durations, counts and approximate throughput', () => {
|
||||
expect(formatTelemetryDuration(0)).toBe('0.0s');
|
||||
expect(formatTelemetryDuration(1234)).toBe('1.2s');
|
||||
expect(formatTelemetryDuration(84000)).toBe('1m24s');
|
||||
expect(formatTelemetryTokens(0)).toBe('0');
|
||||
expect(formatTelemetryTokens(500)).toBe('500');
|
||||
expect(formatTelemetryTokens(1234)).toBe('1.2K');
|
||||
expect(formatTelemetryTokens(1500000)).toBe('1.5M');
|
||||
expect(formatThroughputRate(52.3)).toBe('~52 tok/s');
|
||||
});
|
||||
|
||||
test('aggregates a multi-step turn, subtracting the tool union and including reasoning tokens', () => {
|
||||
const records = turn(assistant({
|
||||
time: { created: 10000, completed: 20000 }, cost: 0.01,
|
||||
tokens: { input: 1000, output: 200, reasoning: 300, cache: { read: 2000, write: 0 } },
|
||||
}), [text(11500), tool(13000, 15000), tool(14000, 16000)]);
|
||||
records.push({ info: assistant({ id: 'a2', time: { created: 21000, completed: 24000 }, cost: 0.005,
|
||||
tokens: { input: 1500, output: 100, reasoning: 0, cache: { read: 0, write: 0 } },
|
||||
}), parts: [text(21500)] });
|
||||
const stats = getLatestCompletedTurnStats(records);
|
||||
expect(stats).toEqual({ stepsCount: 2, lastAssistantMessageId: 'a2', totalToolDurationMs: 3000,
|
||||
totalLlmDurationMs: 10000, outputTokens: 300, reasoningTokens: 300, totalGeneratedTokens: 600,
|
||||
inputTokens: 2500, cost: 0.015, tokensPerSecond: 60, responseTokensPerSecond: null, avgTtftMs: 1000, cacheHitPercent: 44 });
|
||||
});
|
||||
|
||||
test('uses only the latest user-bounded turn', () => {
|
||||
const records = [...turn(), ...turn(assistant({ id: 'new' }))];
|
||||
expect(getLatestCompletedTurnStats(records)?.stepsCount).toBe(1);
|
||||
expect(getLatestCompletedTurnStats(records)?.lastAssistantMessageId).toBe('new');
|
||||
});
|
||||
|
||||
test('does not publish unfinished or truncated turns, or substitute older results', () => {
|
||||
expect(getLatestCompletedTurnStats(null)).toBeNull();
|
||||
expect(getLatestCompletedTurnStats([])).toBeNull();
|
||||
expect(getLatestCompletedTurnStats([{ info: assistant(), parts: [] }])).toBeNull();
|
||||
expect(getLatestCompletedTurnStats([...turn(), { info: user, parts: [] }])).toBeNull();
|
||||
expect(getLatestCompletedTurnStats([...turn(), ...turn(assistant({ time: { created: 1000 } }))])).toBeNull();
|
||||
expect(getLatestCompletedTurnStats([
|
||||
...turn(assistant({ time: { created: 1000 } })), { info: assistant({ id: 'a2' }), parts: [] },
|
||||
])).toBeNull();
|
||||
});
|
||||
|
||||
test('recomputes after history materializes and after same-ID message or part corrections', () => {
|
||||
const info = assistant();
|
||||
expect(getLatestCompletedTurnStats([{ info, parts: [] }])).toBeNull();
|
||||
expect(getLatestCompletedTurnStats(turn(info))?.tokensPerSecond).toBe(25);
|
||||
expect(getLatestCompletedTurnStats(turn({ ...info, tokens: { ...info.tokens, output: 200 } }))?.tokensPerSecond).toBe(50);
|
||||
expect(getLatestCompletedTurnStats(turn(info, [tool(2000, 4000)]))?.tokensPerSecond).toBe(50);
|
||||
// A second directory/runtime may reuse IDs but must never reuse the result.
|
||||
expect(getLatestCompletedTurnStats(turn(info))?.tokensPerSecond).toBe(25);
|
||||
});
|
||||
|
||||
test('missing usage in one step invalidates whole-turn usage, not valid durations', () => {
|
||||
const missing = assistant({ id: 'a2', time: { created: 5000, completed: 6000 } });
|
||||
Reflect.deleteProperty(missing, 'tokens');
|
||||
Reflect.deleteProperty(missing, 'cost');
|
||||
const stats = getLatestCompletedTurnStats([...turn(), { info: missing, parts: [] }]);
|
||||
expect(stats?.stepsCount).toBe(2);
|
||||
expect(stats?.totalLlmDurationMs).toBe(5000);
|
||||
expect(stats?.tokensPerSecond).toBeNull();
|
||||
expect(stats?.inputTokens).toBeNull();
|
||||
expect(stats?.cost).toBeNull();
|
||||
});
|
||||
|
||||
test('missing reasoning is not treated as zero and invalid token counts are not summed', () => {
|
||||
const info = assistant();
|
||||
Reflect.deleteProperty(info.tokens, 'reasoning');
|
||||
expect(getLatestCompletedTurnStats(turn(info))?.tokensPerSecond).toBeNull();
|
||||
for (const output of [-1, NaN, Infinity]) {
|
||||
expect(getLatestCompletedTurnStats(turn(assistant({ tokens: { ...assistant().tokens, output } })))?.totalGeneratedTokens).toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
test('preserves genuine zero usage, cache hits and cost', () => {
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant({ tokens: { ...assistant().tokens, output: 0 } })));
|
||||
expect(stats?.tokensPerSecond).toBe(0);
|
||||
expect(stats?.cost).toBe(0);
|
||||
expect(stats?.cacheHitPercent).toBe(0);
|
||||
});
|
||||
|
||||
test('includes failed tools and chooses the earliest text or reasoning timestamp', () => {
|
||||
const failed: Part = { id: 'failed', sessionID: user.sessionID, messageID: 'a1', type: 'tool', tool: 'bash', callID: 'failed',
|
||||
state: { status: 'error', input: {}, error: 'failed', time: { start: 2500, end: 4000 } } };
|
||||
const reasoning: Part = { id: 'reasoning', sessionID: user.sessionID, messageID: 'a1', type: 'reasoning', text: '', time: { start: 1200 } };
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant(), [text(1600), reasoning, tool(2000, 3000), failed]));
|
||||
expect(stats?.totalToolDurationMs).toBe(2000);
|
||||
expect(stats?.totalLlmDurationMs).toBe(2000);
|
||||
expect(stats?.avgTtftMs).toBe(200);
|
||||
});
|
||||
|
||||
for (const [start, end] of [[0, 2000], [2000, 6000], [3000, 2000], [NaN, 3000]]) {
|
||||
test(`invalid tool interval ${start}..${end} omits duration-dependent metrics`, () => {
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant(), [tool(start, end)]));
|
||||
expect(stats?.totalToolDurationMs).toBeNull();
|
||||
expect(stats?.totalLlmDurationMs).toBeNull();
|
||||
expect(stats?.tokensPerSecond).toBeNull();
|
||||
expect(stats?.outputTokens).toBe(100);
|
||||
});
|
||||
}
|
||||
|
||||
test('unfinished tools and missing tool timing cannot produce a rate', () => {
|
||||
const unfinished: Part = { id: 'pending', sessionID: user.sessionID, messageID: 'a1', type: 'tool', tool: 'bash', callID: 'pending',
|
||||
state: { status: 'pending', input: {}, raw: '' } };
|
||||
const missing = tool(2000, 3000);
|
||||
if (missing.type !== 'tool') throw new Error('Expected tool fixture');
|
||||
Reflect.deleteProperty(missing.state, 'time');
|
||||
expect(getLatestCompletedTurnStats(turn(assistant(), [unfinished]))?.tokensPerSecond).toBeNull();
|
||||
expect(getLatestCompletedTurnStats(turn(assistant(), [missing]))?.tokensPerSecond).toBeNull();
|
||||
});
|
||||
|
||||
test('invalid step time does not silently remove that step from totals', () => {
|
||||
const stats = getLatestCompletedTurnStats([...turn(), { info: assistant({ id: 'a2', time: { created: 6000, completed: 5000 } }), parts: [] }]);
|
||||
expect(stats?.stepsCount).toBe(2);
|
||||
expect(stats?.totalGeneratedTokens).toBe(200);
|
||||
expect(stats?.totalLlmDurationMs).toBeNull();
|
||||
expect(stats?.tokensPerSecond).toBeNull();
|
||||
});
|
||||
|
||||
test('separates final text delivery from whole-turn throughput on the measured tool-heavy shape', () => {
|
||||
const records = turn(assistant({
|
||||
time: { created: 1000, completed: 38438 },
|
||||
tokens: { ...assistant().tokens, output: 223 },
|
||||
}), [tool(19950, 38438)]);
|
||||
records.push({ info: assistant({ id: 'final', time: { created: 40000, completed: 45598 },
|
||||
tokens: { ...assistant().tokens, output: 338 },
|
||||
}), parts: [{ ...text(42661), text: 'Final answer', time: { start: 42661, end: 45442 } }] });
|
||||
const stats = getLatestCompletedTurnStats(records);
|
||||
expect(Math.round(stats?.tokensPerSecond ?? 0)).toBe(23);
|
||||
expect(Math.round(stats?.responseTokensPerSecond ?? 0)).toBe(122);
|
||||
});
|
||||
|
||||
test('measures the final text only, excluding reasoning tokens and their time', () => {
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant({ tokens: { ...assistant().tokens, output: 260, reasoning: 100 } }), [
|
||||
{ id: 'reasoning', sessionID: user.sessionID, messageID: 'a1', type: 'reasoning', text: 'Thinking', time: { start: 1200, end: 2000 } },
|
||||
{ ...text(2500), text: 'Final answer', time: { start: 2500, end: 4500 } },
|
||||
]));
|
||||
expect(stats?.responseTokensPerSecond).toBe(130);
|
||||
expect(stats?.tokensPerSecond).toBe(90);
|
||||
});
|
||||
|
||||
test('unions overlapping text intervals without mutating the authoritative parts', () => {
|
||||
const parts = [
|
||||
{ ...text(2000), text: 'First', time: { start: 2000, end: 3500 } },
|
||||
{ ...text(3000), text: 'Second', time: { start: 3000, end: 4000 } },
|
||||
];
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant(), parts));
|
||||
expect(stats?.responseTokensPerSecond).toBe(50);
|
||||
expect(parts[0].time.end).toBe(3500);
|
||||
});
|
||||
|
||||
test('missing, partial or invalid response timing never falls back to whole-turn speed', () => {
|
||||
const invalidParts: Part[][] = [
|
||||
[], [{ ...text(2000), text: 'No end' }],
|
||||
[{ ...text(2000), text: 'Bad end', time: { start: 2000, end: 1000 } }],
|
||||
[{ ...text(2000), text: 'Late end', time: { start: 2000, end: 6000 } }],
|
||||
[{ ...text(2000), text: 'Zero span', time: { start: 2000, end: 2000 } }],
|
||||
[{ ...text(2000), text: 'Bad time', time: { start: NaN, end: 4000 } }],
|
||||
[{ ...text(2000), text: 'Synthetic', synthetic: true, time: { start: 2000, end: 4000 } }],
|
||||
[{ ...text(2000), text: 'Tool preface', time: { start: 2000, end: 3000 } }, tool(3000, 4000)],
|
||||
[{ ...text(2000), text: 'Timed', time: { start: 2000, end: 3000 } }, { ...text(3000), text: 'Untimed' }],
|
||||
];
|
||||
for (const parts of invalidParts) {
|
||||
const stats = getLatestCompletedTurnStats(turn(assistant(), parts));
|
||||
expect(stats?.responseTokensPerSecond).toBeNull();
|
||||
expect(stats?.tokensPerSecond !== null).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('response speed needs valid output usage and a successful final reply', () => {
|
||||
const parts = [{ ...text(2000), text: 'Final reply', time: { start: 2000, end: 4000 } }];
|
||||
const missingUsage = assistant();
|
||||
Reflect.deleteProperty(missingUsage.tokens, 'output');
|
||||
expect(getLatestCompletedTurnStats(turn(missingUsage, parts))?.responseTokensPerSecond).toBeNull();
|
||||
expect(getLatestCompletedTurnStats(turn(assistant({ error: { name: 'MessageAbortedError', data: { message: 'Stopped' } } }), parts))?.responseTokensPerSecond).toBeNull();
|
||||
expect(getLatestCompletedTurnStats(turn(assistant({ time: { created: NaN, completed: 5000 } }), parts))?.responseTokensPerSecond).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,291 @@
|
||||
import type { Message, Part } from '@opencode-ai/sdk/v2';
|
||||
import { computeCacheHitRate } from '@/stores/utils/tokenUtils';
|
||||
|
||||
type SessionMessageRecord = {
|
||||
info: Message;
|
||||
parts: Part[];
|
||||
};
|
||||
|
||||
type CompletedStepStats = {
|
||||
toolDurationMs: number | null;
|
||||
adjustedLlmDurationMs: number | null;
|
||||
ttftMs: number | null;
|
||||
inputTokens: number | null;
|
||||
outputTokens: number | null;
|
||||
reasoningTokens: number | null;
|
||||
cacheReadTokens: number | null;
|
||||
cacheWriteTokens: number | null;
|
||||
cost: number | null;
|
||||
};
|
||||
|
||||
export type CompletedTurnStats = {
|
||||
lastAssistantMessageId: string;
|
||||
stepsCount: number;
|
||||
totalLlmDurationMs: number | null;
|
||||
totalToolDurationMs: number | null;
|
||||
avgTtftMs: number | null;
|
||||
tokensPerSecond: number | null;
|
||||
responseTokensPerSecond: number | null;
|
||||
inputTokens: number | null;
|
||||
outputTokens: number | null;
|
||||
reasoningTokens: number | null;
|
||||
totalGeneratedTokens: number | null;
|
||||
cacheHitPercent: number | null;
|
||||
cost: number | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Merge an array of [start, end] time intervals into a disjoint union of intervals.
|
||||
* Correctly accounts for parallel / overlapping tool executions without double-counting.
|
||||
*/
|
||||
export function mergeTimeIntervals(intervals: readonly (readonly [number, number])[]): Array<[number, number]> {
|
||||
if (intervals.length === 0) return [];
|
||||
|
||||
const valid: Array<[number, number]> = [];
|
||||
for (const [start, end] of intervals) {
|
||||
if (Number.isFinite(start) && Number.isFinite(end) && end >= start) {
|
||||
valid.push([start, end]);
|
||||
}
|
||||
}
|
||||
|
||||
valid.sort((a, b) => a[0] - b[0]);
|
||||
if (valid.length === 0) return [];
|
||||
|
||||
const merged: Array<[number, number]> = [valid[0]];
|
||||
|
||||
for (let i = 1; i < valid.length; i += 1) {
|
||||
const current = valid[i];
|
||||
const last = merged[merged.length - 1];
|
||||
|
||||
if (current[0] <= last[1]) {
|
||||
last[1] = Math.max(last[1], current[1]);
|
||||
} else {
|
||||
merged.push(current);
|
||||
}
|
||||
}
|
||||
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sum the total duration spanned by an array of disjoint intervals.
|
||||
*/
|
||||
export function sumIntervalsDuration(intervals: readonly (readonly [number, number])[]): number {
|
||||
return intervals.reduce((sum, [start, end]) => sum + (end - start), 0);
|
||||
}
|
||||
|
||||
export const formatTelemetryDuration = (ms: number): string => {
|
||||
if (!Number.isFinite(ms) || ms <= 0) {
|
||||
return '0.0s';
|
||||
}
|
||||
if (ms < 60_000) {
|
||||
return `${(ms / 1000).toFixed(1)}s`;
|
||||
}
|
||||
const minutes = Math.floor(ms / 60_000);
|
||||
const seconds = Math.floor((ms % 60_000) / 1000);
|
||||
return `${minutes}m${seconds}s`;
|
||||
};
|
||||
|
||||
export const formatTelemetryTokens = (tokens: number): string => {
|
||||
if (!Number.isFinite(tokens) || tokens <= 0) {
|
||||
return '0';
|
||||
}
|
||||
if (tokens >= 1_000_000) {
|
||||
return `${(tokens / 1_000_000).toFixed(1)}M`;
|
||||
}
|
||||
if (tokens >= 1_000) {
|
||||
return `${(tokens / 1_000).toFixed(1)}K`;
|
||||
}
|
||||
return String(Math.round(tokens));
|
||||
};
|
||||
|
||||
export const formatThroughputRate = (tps: number): string => {
|
||||
return `~${Math.round(tps)} tok/s`;
|
||||
};
|
||||
|
||||
const nonnegative = (value: number | undefined): number | null =>
|
||||
value !== undefined && Number.isFinite(value) && value >= 0 ? value : null;
|
||||
|
||||
const add = (left: number | null, right: number | null): number | null =>
|
||||
left === null || right === null ? null : nonnegative(left + right);
|
||||
|
||||
/** Text delivery rate for the final reply, not throughput of the agent loop. */
|
||||
function calculateResponseTokenRate(record: SessionMessageRecord): number | null {
|
||||
const { info, parts } = record;
|
||||
if (info.role !== 'assistant' || info.error || parts.some((part) => part.type === 'tool')) return null;
|
||||
const output = nonnegative(info.tokens?.output);
|
||||
const { created, completed } = info.time;
|
||||
if (output === null || completed === undefined || nonnegative(created) === null || nonnegative(completed) === null) return null;
|
||||
|
||||
const intervals: Array<[number, number]> = [];
|
||||
for (const part of parts) {
|
||||
if (part.type !== 'text') continue;
|
||||
// Synthetic/ignored text cannot be matched to the provider's output count.
|
||||
if (part.synthetic || part.ignored) return null;
|
||||
if (!part.text) continue;
|
||||
const start = part.time?.start;
|
||||
const end = part.time?.end;
|
||||
if (start === undefined || end === undefined || !Number.isFinite(start) || !Number.isFinite(end)
|
||||
|| start < created || end > completed || end <= start) return null;
|
||||
intervals.push([start, end]);
|
||||
}
|
||||
const duration = sumIntervalsDuration(mergeTimeIntervals(intervals));
|
||||
return duration > 0 ? nonnegative(output / (duration / 1000)) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculate stats for a single completed assistant step.
|
||||
*/
|
||||
function calculateCompletedStepStats(record: SessionMessageRecord): CompletedStepStats | null {
|
||||
const { info, parts } = record;
|
||||
if (info.role !== 'assistant') return null;
|
||||
|
||||
const { created } = info.time;
|
||||
const completed = info.time.completed;
|
||||
|
||||
if (completed === undefined) return null;
|
||||
|
||||
const validWindow = nonnegative(created) !== null && nonnegative(completed) !== null && completed >= created;
|
||||
const totalDurationMs = validWindow ? nonnegative(completed - created) : null;
|
||||
|
||||
// An unfinished or invalid tool makes duration-dependent metrics unknown.
|
||||
const rawToolIntervals: Array<[number, number]> = [];
|
||||
let validTools = validWindow;
|
||||
for (const part of parts) {
|
||||
if (part.type !== 'tool') continue;
|
||||
if (part.state.status !== 'completed' && part.state.status !== 'error') {
|
||||
validTools = false;
|
||||
continue;
|
||||
}
|
||||
const start = part.state.time?.start;
|
||||
const end = part.state.time?.end;
|
||||
if (!Number.isFinite(start) || !Number.isFinite(end) || start < created || end > completed || end < start) {
|
||||
validTools = false;
|
||||
continue;
|
||||
}
|
||||
rawToolIntervals.push([start, end]);
|
||||
}
|
||||
|
||||
const toolDurationMs = validTools ? nonnegative(sumIntervalsDuration(mergeTimeIntervals(rawToolIntervals))) : null;
|
||||
const adjustedLlmDurationMs = totalDurationMs !== null && toolDurationMs !== null
|
||||
? nonnegative(totalDurationMs - toolDurationMs)
|
||||
: null;
|
||||
|
||||
// Measure TTFT from first text or reasoning part start timestamp
|
||||
let ttftMs: number | null = null;
|
||||
for (const part of parts) {
|
||||
if (part.type === 'text' || part.type === 'reasoning') {
|
||||
const partStart = part.time?.start;
|
||||
if (validWindow && partStart !== undefined && Number.isFinite(partStart) && partStart >= created && partStart <= completed) {
|
||||
const delta = partStart - created;
|
||||
ttftMs = ttftMs === null ? delta : Math.min(ttftMs, delta);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const inputTokens = nonnegative(info.tokens?.input);
|
||||
const outputTokens = nonnegative(info.tokens?.output);
|
||||
const reasoningTokens = nonnegative(info.tokens?.reasoning);
|
||||
const cacheReadTokens = nonnegative(info.tokens?.cache?.read);
|
||||
const cacheWriteTokens = nonnegative(info.tokens?.cache?.write);
|
||||
const cost = nonnegative(info.cost);
|
||||
|
||||
return {
|
||||
toolDurationMs,
|
||||
adjustedLlmDurationMs,
|
||||
ttftMs,
|
||||
inputTokens,
|
||||
outputTokens,
|
||||
reasoningTokens,
|
||||
cacheReadTokens,
|
||||
cacheWriteTokens,
|
||||
cost,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculates telemetry metrics for the latest completed turn in the session.
|
||||
* A turn encompasses all assistant steps since the preceding user message up to the final completed assistant step.
|
||||
*/
|
||||
export function getLatestCompletedTurnStats(
|
||||
records: readonly SessionMessageRecord[] | null | undefined,
|
||||
): CompletedTurnStats | null {
|
||||
if (!records || records.length === 0) return null;
|
||||
|
||||
// Only the newest user-bounded turn qualifies. A partial newer turn must not
|
||||
// be published as complete or silently replaced with an older turn's stats.
|
||||
const lastCompletedAssistantIdx = records.length - 1;
|
||||
if (records[lastCompletedAssistantIdx].info.role !== 'assistant') return null;
|
||||
let turnStartIdx = -1;
|
||||
for (let i = records.length - 1; i >= 0; i -= 1) {
|
||||
const record = records[i];
|
||||
if (record.info.role === 'user') {
|
||||
turnStartIdx = i + 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (turnStartIdx === -1) return null;
|
||||
|
||||
const stepStatsList: CompletedStepStats[] = [];
|
||||
for (let i = turnStartIdx; i <= lastCompletedAssistantIdx; i += 1) {
|
||||
const record = records[i];
|
||||
if (record.info.role === 'assistant') {
|
||||
const stepStats = calculateCompletedStepStats(record);
|
||||
if (!stepStats) return null;
|
||||
stepStatsList.push(stepStats);
|
||||
}
|
||||
}
|
||||
|
||||
if (stepStatsList.length === 0) return null;
|
||||
|
||||
let totalLlmDurationMs: number | null = 0;
|
||||
let totalToolDurationMs: number | null = 0;
|
||||
let totalInputTokens: number | null = 0;
|
||||
let totalOutputTokens: number | null = 0;
|
||||
let totalReasoningTokens: number | null = 0;
|
||||
let totalCacheReadTokens: number | null = 0;
|
||||
let totalCacheWriteTokens: number | null = 0;
|
||||
let totalCost: number | null = 0;
|
||||
let totalTtft: number | null = 0;
|
||||
|
||||
for (const step of stepStatsList) {
|
||||
totalLlmDurationMs = add(totalLlmDurationMs, step.adjustedLlmDurationMs);
|
||||
totalToolDurationMs = add(totalToolDurationMs, step.toolDurationMs);
|
||||
totalInputTokens = add(totalInputTokens, step.inputTokens);
|
||||
totalOutputTokens = add(totalOutputTokens, step.outputTokens);
|
||||
totalReasoningTokens = add(totalReasoningTokens, step.reasoningTokens);
|
||||
totalCacheReadTokens = add(totalCacheReadTokens, step.cacheReadTokens);
|
||||
totalCacheWriteTokens = add(totalCacheWriteTokens, step.cacheWriteTokens);
|
||||
totalCost = add(totalCost, step.cost);
|
||||
totalTtft = add(totalTtft, step.ttftMs);
|
||||
}
|
||||
|
||||
const avgTtftMs = totalTtft === null ? null : totalTtft / stepStatsList.length;
|
||||
|
||||
const totalGeneratedTokens = add(totalOutputTokens, totalReasoningTokens);
|
||||
const tokensPerSecond = totalGeneratedTokens !== null && totalLlmDurationMs !== null && totalLlmDurationMs > 0
|
||||
? nonnegative(totalGeneratedTokens / (totalLlmDurationMs / 1000))
|
||||
: null;
|
||||
|
||||
const cacheHit = totalInputTokens !== null && totalCacheReadTokens !== null && totalCacheWriteTokens !== null ? computeCacheHitRate({
|
||||
input: totalInputTokens,
|
||||
cache: { read: totalCacheReadTokens, write: totalCacheWriteTokens },
|
||||
}) : null;
|
||||
|
||||
return {
|
||||
lastAssistantMessageId: records[lastCompletedAssistantIdx].info.id,
|
||||
stepsCount: stepStatsList.length,
|
||||
totalLlmDurationMs,
|
||||
totalToolDurationMs,
|
||||
avgTtftMs,
|
||||
tokensPerSecond,
|
||||
responseTokensPerSecond: calculateResponseTokenRate(records[lastCompletedAssistantIdx]),
|
||||
inputTokens: totalInputTokens,
|
||||
outputTokens: totalOutputTokens,
|
||||
reasoningTokens: totalReasoningTokens,
|
||||
totalGeneratedTokens,
|
||||
cacheHitPercent: cacheHit?.hasInput ? Math.round(cacheHit.percent) : null,
|
||||
cost: totalCost,
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user