Skip to content

fix(evalboard): variant-run coverage, averaged repeat stats, and per-task model name - #67

Draft
CarlesUIPath wants to merge 4 commits into
mainfrom
fix/evalboard-multimodel-and-model-tag
Draft

fix(evalboard): variant-run coverage, averaged repeat stats, and per-task model name#67
CarlesUIPath wants to merge 4 commits into
mainfrom
fix/evalboard-multimodel-and-model-tag

Conversation

@CarlesUIPath

Copy link
Copy Markdown
Contributor

Summary

This PR implements 2 fixes and proposes a small feature/addon.

  1. Evalboard has no coverage for variant type runs. Current version shows no information:
Screenshot 2026-07-30 at 14 38 00
  1. Evalboard repeats type runs show stats of the first run, when it should show the average (cost, tokens, wall clock time, turns).

  2. Evalboard shows no model name or id within a task, which can become problematic when combining several models/versions across runs and tasks.

Fixes

Variant runs coverage

Screenshot 2026-07-30 at 14 36 40

Evalboard repeats

Now correctly displaying average stats
Screenshot 2026-07-30 at 14 37 33

Model name display

Now displaying name next to task verdict
Screenshot 2026-07-30 at 14 35 49

Root cause (variant runs)

The dashboard had no concept of an experiment variant. In an A/B run each model produces a task_result sharing the task_id, so it broke on two surfaces:

  • Grid — the replicate collapse was keyed on task_id alone, folding every model into one row with metrics averaged across models; the distinct models were never shown.
  • Detail — the per-task content path was hardcoded to a default/ subdir, which doesn't exist for a named variant (kimi-k3/…), so opening any multi-model task 404'd.

Changes

Variant is now a first-class dimension, end to end:

  • Data model (lib/runs.ts) — capture run.json variant_id as TaskResultSummary.variant (null on legacy rows → treated as "default").
  • Paths — thread a sanitized variant (default "default") through taskContentBase, ensureTaskDir (lib/blob.ts), readTaskDetail (now also matches the row on variant), readTaskReplicates, readLogTail, readConversationLog, collectTaskFiles, resolveSafePath, and the download API.
  • Grid (lib/status.ts, task-grid.tsx) — collapse + pass-counts key on (taskId, variant) via a new taskGroupKey, so each model keeps its own row and its own metrics. New Model column, shown only when a run has >1 distinct model. Detail links carry ?v=<variant>.
  • Detail page ([...task]/page.tsx) — parse ?v=, thread it to every reader, preserve it on the replicate selector + download link, and show the model chip in the header (fix chore: Bump actions/cache from 5.0.5 to 6.1.0 #3) so a task always names the LLM it ran on — single-model runs included.
  • Repeats (fix chore: Bump actions/setup-node from 4.4.0 to 6.4.0 #2) — collapseReplicates now averages the quantitative columns (score, duration, cost, turns, tokens) across a task's replicates instead of showing the first/representative run's values.

Variant ids are validated as a single path segment (they land in filesystem paths and blob prefixes); anything unsafe or unknown falls back to "default".

Backward compatibility

Single-config runs carry variant "default", so their grid, links, and paths are byte-for-behavior unchanged — verified live: no Model column, no ?v=, model chip still shown.

Testing

Result: all suites greentsc --noEmit clean, next build succeeds, 363/365 unit tests pass. The only 2 failures are a pre-existing, unrelated pricing-table parity check (see Not in this PR), which fails identically on the base branch. 19 new tests were added for this change; all pass.

  • Logic: variant grouping keeps one row per model, taskGroupKey collision-safety, perTaskPassCounts per-variant, grid Model-column show/hide + ?v= links, toTaskRow variant/model mapping.
  • Path resolution (lib/__tests__/variant-paths.test.ts, against a temp local runs dir): readTaskDetail selects the right model's row (not the first variant) and returns null / sanitizes an unsafe ?v; readTaskReplicates / readLogTail / collectTaskFiles are scoped to the selected variant; resolveSafePath resolves a non-default variant artifact and rejects traversal. This suite caught (and now guards) a regression where a broadened resolveSafePath handed a .. segment to ensureTaskDir and threw instead of returning null — fixed by restricting the narrow-fetch prefetch to safe segments.
  • Live E2E against a real 3-model run (kimi-k3 / glm-5.2 / deepseek-v4-pro): all three render as distinct rows with the Model column; each links to its own ?v=; all three per-model detail pages load (HTTP 200) with the correct model chip — these all 404'd before. Single-model regression clean.

Known coverage gap

The task-detail page ([...task]/page.tsx) is a React Server Component and has no unit test for its ?v= parsing / model-chip render — there's no RSC test harness in the repo. That logic is simple and is covered by the live E2E above.

Not in this PR

  • Pre-existing, unrelated failure in lib/__tests__/pricing-parity.test.ts (2 cases — claude-sonnet-5, gpt-5.6-terra missing from lib/pricing.ts). Confirmed failing identically before these changes.

CarlesUIPath and others added 4 commits July 27, 2026 18:23
The run page task-grid collapsed a task's replicates to a single
representative row and displayed that one run's metrics verbatim, so the
Cost column (and score/duration/turns/tokens) showed the representative
replicate's value instead of the average over the repeats.

Add collapseReplicates() to lib/status.ts: it keeps the representative
only for categorical fields (status pill, ?r=NN detail link, tags/skill/
model) and averages the quantitative columns across all replicates. With
repeats disabled it's a no-op. Round the now-fractional turns display to
at most 2 decimals (dropping trailing zeros) in fmtTurnsCount.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…odel runs

The dashboard had no concept of an experiment *variant*, so A/B (multi-model)
runs were mis-rendered on two surfaces:

  - Grid: each model produces a task_result sharing the task_id, so the
    replicate collapse (keyed on task_id alone) folded every model into one
    row with metrics averaged ACROSS models — the distinct models weren't
    shown at all.
  - Detail: the per-task content path was hardcoded to a `default/` subdir,
    which doesn't exist for a named variant (`kimi-k3/…`), so opening any
    multi-model task 404'd.

Make the variant a first-class dimension end to end:

  - Data model: capture run.json `variant_id` as TaskResultSummary.variant
    (null on legacy rows → treated as "default").
  - Paths: thread a sanitized `variant` (default "default") through
    taskContentBase, ensureTaskDir, readTaskDetail (now also matches the row
    on variant), readTaskReplicates, readLogTail, readConversationLog,
    collectTaskFiles, resolveSafePath, and the download API.
  - Grid: collapse + pass-counts key on (taskId, variant) via a new
    taskGroupKey, so each model keeps its own row and its own metrics. Add a
    Model column shown only when a run has >1 distinct model, and carry the
    variant on detail links (?v=).
  - Detail page: parse ?v=, thread it to every reader, preserve it on the
    replicate selector + download link, and show a model chip in the header
    so a task always names the LLM it ran on (single-model runs included).

Single-config runs carry variant "default", so their grid, links and paths
are byte-for-behavior unchanged (verified: no Model column, no ?v=, model
chip still shown). Tests: variant grouping / taskGroupKey collision-safety /
perTaskPassCounts, grid Model-column + ?v= link rendering, toTaskRow mapping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h traversal

Add lib/__tests__/variant-paths.test.ts exercising the variant-aware readers
against a temp local runs dir (the collect.test.ts env-stub + fresh-import
harness): readTaskDetail picks the right model's row (not the first variant),
returns null with no ?v on a run that has no "default" subdir, and sanitizes an
unsafe ?v to "default"; readTaskReplicates / readLogTail / collectTaskFiles are
scoped to the selected variant; resolveSafePath resolves a non-default variant
artifact and rejects traversal.

That last test caught a regression: the broadened resolveSafePath handed a
"../.." segment to ensureTaskDir, which THROWS (isValidId admits dots), so the
/api/file route would 500 instead of 403. Restrict the narrow-fetch prefetch to
genuinely safe segments (reject "."/".."); traversal now falls through to the
realpath containment check and returns null as before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant