fix(evalboard): variant-run coverage, averaged repeat stats, and per-task model name - #67
Draft
CarlesUIPath wants to merge 4 commits into
Draft
fix(evalboard): variant-run coverage, averaged repeat stats, and per-task model name#67CarlesUIPath wants to merge 4 commits into
CarlesUIPath wants to merge 4 commits into
Conversation
The run page task-grid collapsed a task's replicates to a single representative row and displayed that one run's metrics verbatim, so the Cost column (and score/duration/turns/tokens) showed the representative replicate's value instead of the average over the repeats. Add collapseReplicates() to lib/status.ts: it keeps the representative only for categorical fields (status pill, ?r=NN detail link, tags/skill/ model) and averages the quantitative columns across all replicates. With repeats disabled it's a no-op. Round the now-fractional turns display to at most 2 decimals (dropping trailing zeros) in fmtTurnsCount. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…odel runs
The dashboard had no concept of an experiment *variant*, so A/B (multi-model)
runs were mis-rendered on two surfaces:
- Grid: each model produces a task_result sharing the task_id, so the
replicate collapse (keyed on task_id alone) folded every model into one
row with metrics averaged ACROSS models — the distinct models weren't
shown at all.
- Detail: the per-task content path was hardcoded to a `default/` subdir,
which doesn't exist for a named variant (`kimi-k3/…`), so opening any
multi-model task 404'd.
Make the variant a first-class dimension end to end:
- Data model: capture run.json `variant_id` as TaskResultSummary.variant
(null on legacy rows → treated as "default").
- Paths: thread a sanitized `variant` (default "default") through
taskContentBase, ensureTaskDir, readTaskDetail (now also matches the row
on variant), readTaskReplicates, readLogTail, readConversationLog,
collectTaskFiles, resolveSafePath, and the download API.
- Grid: collapse + pass-counts key on (taskId, variant) via a new
taskGroupKey, so each model keeps its own row and its own metrics. Add a
Model column shown only when a run has >1 distinct model, and carry the
variant on detail links (?v=).
- Detail page: parse ?v=, thread it to every reader, preserve it on the
replicate selector + download link, and show a model chip in the header
so a task always names the LLM it ran on (single-model runs included).
Single-config runs carry variant "default", so their grid, links and paths
are byte-for-behavior unchanged (verified: no Model column, no ?v=, model
chip still shown). Tests: variant grouping / taskGroupKey collision-safety /
perTaskPassCounts, grid Model-column + ?v= link rendering, toTaskRow mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h traversal Add lib/__tests__/variant-paths.test.ts exercising the variant-aware readers against a temp local runs dir (the collect.test.ts env-stub + fresh-import harness): readTaskDetail picks the right model's row (not the first variant), returns null with no ?v on a run that has no "default" subdir, and sanitizes an unsafe ?v to "default"; readTaskReplicates / readLogTail / collectTaskFiles are scoped to the selected variant; resolveSafePath resolves a non-default variant artifact and rejects traversal. That last test caught a regression: the broadened resolveSafePath handed a "../.." segment to ensureTaskDir, which THROWS (isValidId admits dots), so the /api/file route would 500 instead of 403. Restrict the narrow-fetch prefetch to genuinely safe segments (reject "."/".."); traversal now falls through to the realpath containment check and returns null as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements 2 fixes and proposes a small feature/addon.
varianttype runs. Current version shows no information:Evalboard
repeatstype runs show stats of the first run, when it should show the average (cost, tokens, wall clock time, turns).Evalboard shows no model name or id within a task, which can become problematic when combining several models/versions across runs and tasks.
Fixes
Variant runs coverage
Evalboard repeats
Now correctly displaying average stats

Model name display
Now displaying name next to task verdict

Root cause (variant runs)
The dashboard had no concept of an experiment variant. In an A/B run each model produces a
task_resultsharing thetask_id, so it broke on two surfaces:task_idalone, folding every model into one row with metrics averaged across models; the distinct models were never shown.default/subdir, which doesn't exist for a named variant (kimi-k3/…), so opening any multi-model task 404'd.Changes
Variant is now a first-class dimension, end to end:
lib/runs.ts) — capture run.jsonvariant_idasTaskResultSummary.variant(null on legacy rows → treated as"default").variant(default"default") throughtaskContentBase,ensureTaskDir(lib/blob.ts),readTaskDetail(now also matches the row on variant),readTaskReplicates,readLogTail,readConversationLog,collectTaskFiles,resolveSafePath, and the download API.lib/status.ts,task-grid.tsx) — collapse + pass-counts key on(taskId, variant)via a newtaskGroupKey, so each model keeps its own row and its own metrics. New Model column, shown only when a run has >1 distinct model. Detail links carry?v=<variant>.[...task]/page.tsx) — parse?v=, thread it to every reader, preserve it on the replicate selector + download link, and show the model chip in the header (fix chore: Bump actions/cache from 5.0.5 to 6.1.0 #3) so a task always names the LLM it ran on — single-model runs included.collapseReplicatesnow averages the quantitative columns (score, duration, cost, turns, tokens) across a task's replicates instead of showing the first/representative run's values.Variant ids are validated as a single path segment (they land in filesystem paths and blob prefixes); anything unsafe or unknown falls back to
"default".Backward compatibility
Single-config runs carry variant
"default", so their grid, links, and paths are byte-for-behavior unchanged — verified live: no Model column, no?v=, model chip still shown.Testing
Result: all suites green —
tsc --noEmitclean,next buildsucceeds, 363/365 unit tests pass. The only 2 failures are a pre-existing, unrelated pricing-table parity check (see Not in this PR), which fails identically on the base branch. 19 new tests were added for this change; all pass.taskGroupKeycollision-safety,perTaskPassCountsper-variant, grid Model-column show/hide +?v=links,toTaskRowvariant/model mapping.lib/__tests__/variant-paths.test.ts, against a temp local runs dir):readTaskDetailselects the right model's row (not the first variant) and returns null / sanitizes an unsafe?v;readTaskReplicates/readLogTail/collectTaskFilesare scoped to the selected variant;resolveSafePathresolves a non-default variant artifact and rejects traversal. This suite caught (and now guards) a regression where a broadenedresolveSafePathhanded a..segment toensureTaskDirand threw instead of returning null — fixed by restricting the narrow-fetch prefetch to safe segments.kimi-k3/glm-5.2/deepseek-v4-pro): all three render as distinct rows with the Model column; each links to its own?v=; all three per-model detail pages load (HTTP 200) with the correct model chip — these all 404'd before. Single-model regression clean.Known coverage gap
The task-detail page (
[...task]/page.tsx) is a React Server Component and has no unit test for its?v=parsing / model-chip render — there's no RSC test harness in the repo. That logic is simple and is covered by the live E2E above.Not in this PR
lib/__tests__/pricing-parity.test.ts(2 cases —claude-sonnet-5,gpt-5.6-terramissing fromlib/pricing.ts). Confirmed failing identically before these changes.