Skip to content

Add [[ablation]] sweep syntax for eval TOML configs#993

Open
snimu wants to merge 4 commits intomainfrom
sebastian/vf-eval-ablations-2026-03-05
Open

Add [[ablation]] sweep syntax for eval TOML configs#993
snimu wants to merge 4 commits intomainfrom
sebastian/vf-eval-ablations-2026-03-05

Conversation

@snimu
Copy link
Contributor

@snimu snimu commented Mar 5, 2026

Expand cartesian products of parameter values into eval configs automatically. Supports top-level sweep keys and env_args sweep keys, fixed env_args alongside swept ones (with overlap guard), multiple independent ablation blocks, and coexistence with [[eval]].

Description

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Test improvement

Testing

  • All existing tests pass when running uv run pytest locally.
  • New tests have been added to cover the changes

Checklist

  • My code follows the style guidelines of this project as outlined in AGENTS.md
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Note

Medium Risk
Touches core config-loading logic used by prime eval, so schema/validation mistakes could break existing TOML workflows, though changes are well-scoped and covered by new tests.

Overview
Adds support for [[ablation]] blocks in eval TOML configs, expanding a cartesian product of [ablation.sweep] (and [ablation.sweep.env_args]) values into multiple concrete eval configs alongside existing [[eval]] entries.

Updates load_toml_config to parse/validate ablation sections, apply global defaults to generated configs, guard against invalid fields and overlapping fixed vs swept env_args, and continue resolving endpoints_path relative to the config. Documentation and the eval skill guide now describe the new syntax, and the test suite adds comprehensive coverage for expansion behavior and error cases.

Written by Cursor Bugbot for commit 4d8dd11. This will update automatically on new commits. Configure here.

snimu and others added 2 commits March 5, 2026 14:13
Expand cartesian products of parameter values into eval configs
automatically. Supports top-level sweep keys and env_args sweep
keys, fixed env_args alongside swept ones (with overlap guard),
multiple independent ablation blocks, and coexistence with [[eval]].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

snimu and others added 2 commits March 5, 2026 14:25
Shallow copy of fixed dict caused all expanded configs to share
the same env_args reference. Now dict values are copied per config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@snimu snimu requested review from mikasenghaas and willccbb March 9, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant