Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions apps/dashboard/src/components/AnalyticsCharts.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ export function AnalyticsCharts({ data, projectId }: AnalyticsChartsProps) {
htmlFor="baseline-select"
className="text-xs font-medium uppercase tracking-wider text-gray-500"
>
Baseline target
Baseline provider/model
</label>
<select
id="baseline-select"
Expand Down Expand Up @@ -158,7 +158,7 @@ export function AnalyticsCharts({ data, projectId }: AnalyticsChartsProps) {

{!baseline && (
<p className="text-xs text-gray-500">
Select a baseline target above to see gain and delta charts.
Select a baseline provider/model above to see gain and delta charts.
</p>
)}
</div>
Expand Down Expand Up @@ -237,7 +237,7 @@ function NormalizedGainChart({ data, baseline }: { data: CompareResponse; baseli
return (
<ChartSection title="Normalized Gain (g)">
<p className="text-xs text-gray-500">
No gain data available. Ensure multiple targets exist.
No gain data available. Ensure multiple provider/model labels exist.
</p>
</ChartSection>
);
Expand Down Expand Up @@ -313,7 +313,7 @@ function NegativeDeltaTable({ data, baseline }: { data: CompareResponse; baselin
<thead className="border-b border-gray-800 bg-gray-900/50">
<tr>
<th className="px-4 py-3 font-medium text-gray-400">Experiment</th>
<th className="px-4 py-3 font-medium text-gray-400">Target</th>
<th className="px-4 py-3 font-medium text-gray-400">Provider/Model</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Avg Score</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Delta</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">g</th>
Expand Down
2 changes: 1 addition & 1 deletion apps/dashboard/src/components/EvalDetail.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ function SourceTab({ result }: { result: EvalResult }) {
<SourceMetaRow label="Test ID" value={traceability.test_id ?? result.testId} />
<SourceMetaRow label="Legacy suite" value={result.suite} />
<SourceMetaRow label="Category" value={result.category} />
<SourceMetaRow label="Target" value={result.target} />
<SourceMetaRow label="Provider/Model" value={result.target} />
</dl>
</section>

Expand Down
4 changes: 2 additions & 2 deletions apps/dashboard/src/components/ResultTable.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ describe('ResultTable target error kind', () => {
target_error_kind: 'timeout',
});

expect(html).toContain('[target:timeout] target timed out');
expect(html).toContain('[provider:timeout] target timed out');
});

it('falls back to the legacy nested target_execution shape on older bundles', () => {
Expand All @@ -145,6 +145,6 @@ describe('ResultTable target error kind', () => {
target_execution: { error_kind: 'timeout' },
});

expect(html).toContain('[target:timeout] target timed out');
expect(html).toContain('[provider:timeout] target timed out');
});
});
10 changes: 5 additions & 5 deletions apps/dashboard/src/components/ResultTable.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -345,19 +345,19 @@ export function ResultTable({
type="search"
value={model.state.search}
onChange={(event) => updateState({ search: event.target.value })}
placeholder="Search tests, targets, graders, assertions"
placeholder="Search tests, providers, models, graders, assertions"
className="w-full rounded-md border border-gray-700 bg-gray-950 px-3 py-1.5 text-sm text-gray-100 placeholder:text-gray-600 focus:border-cyan-500 focus:outline-none focus:ring-1 focus:ring-cyan-500"
/>
</label>

<label>
<span className="sr-only">Filter by target</span>
<span className="sr-only">Filter by provider/model</span>
<select
value={model.state.target}
onChange={(event) => updateState({ target: event.target.value })}
className="w-full rounded-md border border-gray-700 bg-gray-950 px-3 py-1.5 text-sm text-gray-100 focus:border-cyan-500 focus:outline-none focus:ring-1 focus:ring-cyan-500"
>
<option value="all">All targets</option>
<option value="all">All provider/model labels</option>
{model.targetOptions.map((target) => (
<option key={target} value={target}>
{target}
Expand Down Expand Up @@ -642,8 +642,8 @@ function formatTargetError(
error: string | undefined,
kind: string | undefined,
): string | undefined {
if (!error) return kind ? `Target error: ${kind}` : undefined;
return kind ? `[target:${kind}] ${error}` : error;
if (!error) return kind ? `Provider error: ${kind}` : undefined;
return kind ? `[provider:${kind}] ${error}` : error;
}

function targetErrorKind(value: {
Expand Down
6 changes: 3 additions & 3 deletions apps/dashboard/src/components/RunEvalModal.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -294,18 +294,18 @@ export function RunEvalModal({ open, onClose, projectId, prefill }: RunEvalModal
)}
</div>

{/* Target override */}
{/* Provider override */}
<div>
<label htmlFor="target-override" className="mb-1 block text-sm font-medium text-gray-300">
Target Override
Provider Override
</label>
<select
id="target-override"
value={target}
onChange={(e) => setTarget(e.target.value)}
className="w-full rounded-md border border-gray-700 bg-gray-800 px-3 py-2 text-sm text-white focus:border-cyan-600 focus:outline-none"
>
<option value="">Use eval's configured target</option>
<option value="">Use eval's configured provider</option>
{targetNames.map((name) => (
<option key={name} value={name}>
{name}
Expand Down
6 changes: 3 additions & 3 deletions apps/dashboard/src/components/RunList.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ export function RunList({
}
const count = err.duplicates.length;
const confirmed = window.confirm(
`${count} duplicate (test_id, target) pair${count === 1 ? '' : 's'} found. Replace duplicates with the latest timestamp?`,
`${count} duplicate (test ID, provider/model) pair${count === 1 ? '' : 's'} found. Replace duplicates with the latest timestamp?`,
);
if (!confirmed) return;
result = await combineRunsApi(sourceRunIds, 'latest', projectId, undefined, experiment);
Expand Down Expand Up @@ -533,7 +533,7 @@ export function RunList({
{enableCombine && <th className="w-10 px-4 py-3" />}
<th className="w-8 px-4 py-3" />
<th className="w-[18rem] px-4 py-3 font-medium text-gray-400">Experiment</th>
<th className="w-[16rem] px-4 py-3 font-medium text-gray-400">Target</th>
<th className="w-[16rem] px-4 py-3 font-medium text-gray-400">Provider/Model</th>
<th className="px-4 py-3 font-medium text-gray-400">Remote</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Passed</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Failures</th>
Expand Down Expand Up @@ -606,7 +606,7 @@ export function RunList({
</div>
</td>

{/* Target */}
{/* Provider/model label */}
<td className="w-[16rem] max-w-[16rem] px-4 py-3">
<div className="min-w-0">
<div className="flex min-w-0 items-center gap-2">
Expand Down
2 changes: 1 addition & 1 deletion apps/dashboard/src/components/Sidebar.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ const projectNavItems: { id: ProjectTabId; label: string; description: string }[
{ id: 'runs', label: 'Recent Runs', description: 'Run review' },
{ id: 'tags', label: 'Tags', description: 'Grouped runs' },
{ id: 'analytics', label: 'Analytics', description: 'Compare scores' },
{ id: 'targets', label: 'Targets', description: 'Target results' },
{ id: 'targets', label: 'Providers', description: 'Provider results' },
];

function sidebarLinkClass(isActive: boolean): string {
Expand Down
4 changes: 2 additions & 2 deletions apps/dashboard/src/components/TagValueDetail.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ export function TagValueDetail({ tagKey, tagValue, projectId }: TagValueDetailPr
</p>
<h1 className="text-2xl font-semibold text-white">{tagValue}</h1>
<p className="mt-1 text-sm text-gray-400">
{runCount} run{runCount !== 1 ? 's' : ''} &middot; {targetCount} target
{runCount} run{runCount !== 1 ? 's' : ''} &middot; {targetCount} provider/model label
{targetCount !== 1 ? 's' : ''}
{group?.last_run && (
<span className="ml-2">&middot; Last run: {formatTimestamp(group.last_run)}</span>
Expand All @@ -81,7 +81,7 @@ export function TagValueDetail({ tagKey, tagValue, projectId }: TagValueDetailPr
{group && (
<div className="grid grid-cols-2 gap-4 sm:grid-cols-4">
<StatCard label="Runs" value={String(runCount)} />
<StatCard label="Targets" value={String(targetCount)} />
<StatCard label="Providers" value={String(targetCount)} />
<StatCard
label="Pass Rate"
value={`${Math.round(passRate * 100)}%`}
Expand Down
2 changes: 1 addition & 1 deletion apps/dashboard/src/components/TagsTab.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ export function TagsTab({ projectId, tagKey, onTagKeyChange }: TagsTabProps) {
<tr>
<th className="px-4 py-3 font-medium text-gray-400">{tagKeyLabel(effectiveKey)}</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Runs</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Targets</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Providers</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Evals</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Execution Errors</th>
<th className="px-4 py-3 font-medium text-gray-400">Pass Rate</th>
Expand Down
15 changes: 8 additions & 7 deletions apps/dashboard/src/components/TargetsTab.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -95,17 +95,17 @@ export function TargetsTab({ projectId }: TargetsTabProps = {}) {
if (error) {
return (
<div className="rounded-lg border border-red-900/50 bg-red-950/20 p-6 text-red-400">
Failed to load targets: {error.message}
Failed to load provider/model labels: {error.message}
</div>
);
}

if (targets.length === 0) {
return (
<div className="rounded-lg border border-gray-800 bg-gray-900 p-8 text-center">
<p className="text-lg text-gray-400">No targets found</p>
<p className="text-lg text-gray-400">No provider/model labels found</p>
<p className="mt-2 text-sm text-gray-500">
Targets will appear here once evaluations are run with target labels.
Provider/model labels will appear here once evaluations have run.
</p>
</div>
);
Expand All @@ -117,7 +117,7 @@ export function TargetsTab({ projectId }: TargetsTabProps = {}) {
<table className="min-w-[720px] w-full whitespace-nowrap text-left text-sm">
<thead className="border-b border-gray-800 bg-gray-900/50">
<tr>
<th className="px-4 py-3 font-medium text-gray-400">Target</th>
<th className="px-4 py-3 font-medium text-gray-400">Provider/Model</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Runs</th>
<th className="px-4 py-3 text-right font-medium text-gray-400">Experiments</th>
<th className="px-4 py-3 font-medium text-gray-400">Pass Rate</th>
Expand Down Expand Up @@ -174,7 +174,7 @@ export function TargetsTab({ projectId }: TargetsTabProps = {}) {
onClick={() => setSelectedTargetName(null)}
className="rounded-md px-3 py-1.5 text-sm text-gray-400 transition-colors hover:text-gray-200"
>
← Back to targets
← Back to providers
</button>
<div className="rounded-lg border border-gray-800 bg-gray-900/50 p-4">
<div className="flex flex-col gap-3 sm:flex-row sm:items-start sm:justify-between">
Expand Down Expand Up @@ -203,9 +203,10 @@ export function TargetsTab({ projectId }: TargetsTabProps = {}) {

{experimentGroups.length === 0 ? (
<div className="rounded-lg border border-gray-800 bg-gray-900 p-8 text-center">
<p className="text-lg text-gray-400">No runs found for this target</p>
<p className="text-lg text-gray-400">No runs found for this provider/model label</p>
<p className="mt-2 text-sm text-gray-500">
This target summary exists, but there are no matching runs to group by experiment.
This provider/model summary exists, but there are no matching runs to group by
experiment.
</p>
</div>
) : (
Expand Down
1 change: 1 addition & 0 deletions apps/dashboard/src/lib/result-table.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ describe('result-table model', () => {
]);
expect(model.visibleColumns.map((column) => column.id)).toContain('grader:correctness');
expect(model.columns.find((column) => column.id === 'eval')?.label).toBe('Eval');
expect(model.columns.find((column) => column.id === 'target')?.label).toBe('Provider/Model');
expect(model.rows[0].evalLabel).toBe('evals/dataset.eval.yaml');
});

Expand Down
2 changes: 1 addition & 1 deletion apps/dashboard/src/lib/result-table.ts
Original file line number Diff line number Diff line change
Expand Up @@ -368,7 +368,7 @@ function buildColumns(rows: readonly ResultTableRow[], graderOptions: readonly s
? [{ id: 'expander', label: 'Expand', kind: 'base' as const, defaultVisible: true }]
: []),
{ id: 'test', label: 'Test ID', kind: 'base', defaultVisible: true },
{ id: 'target', label: 'Target', kind: 'base', defaultVisible: true },
{ id: 'target', label: 'Provider/Model', kind: 'base', defaultVisible: true },
...(hasEval
? [{ id: 'eval', label: 'Eval', kind: 'base' as const, defaultVisible: true }]
: []),
Expand Down
4 changes: 2 additions & 2 deletions apps/dashboard/src/routes/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ const tabs: { id: TabId; label: string }[] = [
{ id: 'runs', label: '🏃 Recent Runs' },
{ id: 'tags', label: '🏷️ Tags' },
{ id: 'analytics', label: '📊 Analytics' },
{ id: 'targets', label: '🤖 Targets' },
{ id: 'targets', label: '🤖 Providers' },
];

export const Route = createFileRoute('/')({
Expand Down Expand Up @@ -189,7 +189,7 @@ function ProjectsDashboard() {
<div className="rounded-lg border border-gray-800 bg-gray-900/50 p-8 text-center">
<p className="text-lg text-gray-300">No projects registered yet.</p>
<p className="mt-2 text-sm text-gray-500">
Add a project path to start browsing runs, tags, analytics, and targets.
Add a project path to start browsing runs, tags, analytics, and providers.
</p>
</div>
) : (
Expand Down
4 changes: 2 additions & 2 deletions apps/dashboard/src/routes/projects/$projectId.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* Project home route: tabbed view (Runs, Tags, Analytics, Targets) scoped to a project.
* Project home route: tabbed view (Runs, Tags, Analytics, Providers) scoped to a project.
*
* Mirrors the single-project home page but fetches from project-scoped API endpoints.
*/
Expand Down Expand Up @@ -38,7 +38,7 @@ const tabs: { id: TabId; label: string; title: string }[] = [
{ id: 'runs', label: '🏃 Recent Runs', title: 'Recent Runs' },
{ id: 'tags', label: '🏷️ Tags', title: 'Tags' },
{ id: 'analytics', label: '📊 Analytics', title: 'Analytics' },
{ id: 'targets', label: '🤖 Targets', title: 'Targets' },
{ id: 'targets', label: '🤖 Providers', title: 'Providers' },
];

export const Route = createFileRoute('/projects/$projectId')({
Expand Down
14 changes: 7 additions & 7 deletions apps/web/src/content/docs/docs/next/tools/dashboard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import studioRemoteResultsAfterSync from '../../../../../assets/screenshots/stud

The `dashboard` command launches a web-based dashboard for browsing evaluation runs, inspecting individual test results, and reviewing scores. It shows both local runs and runs synced from a remote results repository.

<Image src={studioRunsBench} alt="AgentV Dashboard showing evaluation runs with pass rates, targets, and experiment names" />
<Image src={studioRunsBench} alt="AgentV Dashboard showing evaluation runs with pass rates, provider/model labels, and experiment names" />

## Usage

Expand Down Expand Up @@ -72,7 +72,7 @@ indexes into Phoenix.

- **Recent Runs** — table of all evaluation runs with source badge (`local` / `remote`), provider/model label, experiment, timestamp, test count, pass rate, and mean score
- **Experiments** — group and compare runs by experiment name
- **Targets** — group runs by provider/model label
- **Providers** — group runs by provider/model label
- **Run Detail** — drill into a run to see per-test results, scores, and grader output
- **Analytics** — provider/model comparison for multi-provider eval runs, an aggregated experiment × provider/model matrix, and a per-run view for selecting individual runs to compare side-by-side with optional retroactive tags. Includes a collapsible charts section with baseline comparison analytics
- **Remote Results** — sync and browse runs pushed from other machines or CI (see [Remote Results](#remote-results))
Expand Down Expand Up @@ -124,7 +124,7 @@ The **Analytics** tab has two modes: **Aggregated** for the experiment × provid

The default view shows a cross-experiment, cross-provider/model performance matrix. Numbers are colour-coded by pass rate — green (80%+), amber (50–80%), red (below 50%) — and each cell shows `passed/total` and the mean score. Click any cell to expand the per-test-case breakdown.

<Image src={studioAnalyticsAggregated} alt="AgentV Dashboard Analytics tab showing aggregated experiment × target matrix with pass rates for baseline, optimized-prompt, and with-rag across claude-sonnet, gemini-pro, and gpt-4o" />
<Image src={studioAnalyticsAggregated} alt="AgentV Dashboard Analytics tab showing aggregated experiment by provider/model matrix with pass rates for baseline, optimized-prompt, and with-rag across claude-sonnet, gemini-pro, and gpt-4o" />

Run the same eval against multiple providers or experiment variants, then open the Analytics tab:

Expand Down Expand Up @@ -156,7 +156,7 @@ Keep model identity in `providers` and provider `label`s. Use `tags.experiment`

Running the same `(experiment, provider/model label)` twice no longer collapses into a single cell. Switch to **Per run** mode to see every run as its own row, select two or more, and compare them head-to-head.

<Image src={studioComparePerRun} alt="AgentV Dashboard per-run compare mode with a filter-by-tag chip row and individual runs listing timestamp, tags, experiment, target, and pass rate; experiment-prefixed runs surface the experiment name under the timestamp" />
<Image src={studioComparePerRun} alt="AgentV Dashboard per-run compare mode with a filter-by-tag chip row and individual runs listing timestamp, tags, experiment, provider/model label, and pass rate; experiment-prefixed runs surface the experiment name under the timestamp" />

Use per-run mode when you want to:

Expand Down Expand Up @@ -192,11 +192,11 @@ The section includes the following visualizations:
- **Tag × Provider/Model Heatmap** — pass-rate grid across tags and provider/model labels, colour-coded by performance (emerald for high, amber for medium, red for low).
- **Negative Delta Table** — filtered list of experiment × provider/model pairs that scored worse than the baseline, sorted by largest regression.
- **Score Distribution** — histogram showing the variance of scores across all test cases, binned by 10% intervals.
- **Score Trend Over Time** — line chart plotting mean score per target across runs over time, with a colour-coded legend for each target.
- **Score Trend Over Time** — line chart plotting mean score per provider/model label across runs over time, with a colour-coded legend for each label.

<Image src={studioAnalyticsTrend} alt="AgentV Dashboard analytics showing score distribution histogram and score trend over time line chart with multi-target legend" />
<Image src={studioAnalyticsTrend} alt="AgentV Dashboard analytics showing score distribution histogram and score trend over time line chart with multi-provider/model legend" />

The baseline comparison is also available via the API: `GET /api/compare?baseline=<target>` adds `delta` and `normalized_gain` fields to each non-baseline cell in the response.
The baseline comparison is also available via the API: `GET /api/compare?baseline=<target>` adds `delta` and `normalized_gain` fields to each non-baseline provider/model cell in the response.

## Projects Dashboard

Expand Down
Loading