diff --git a/.codex/agents/modelopt_model_evaluator.toml b/.codex/agents/modelopt_model_evaluator.toml index ad99a342d4b..4a758e3d81a 100644 --- a/.codex/agents/modelopt_model_evaluator.toml +++ b/.codex/agents/modelopt_model_evaluator.toml @@ -13,5 +13,7 @@ Before acting, load these Model Optimizer instructions: Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. +Apply `evaluation/references/mlflow-verification.md` independently of evaluation acceptance. Recover export from existing results only; never submit another evaluation to repair delivery. In `MLflow`, report each task's export outcome and verified run URL (or none verified), mapped to its invocation ID. Keep evaluation and export outcomes separate; report blockers and preserved evidence paths when delivery fails. + Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. """ diff --git a/plugins/modelopt/agents/modelopt-model-evaluator.md b/plugins/modelopt/agents/modelopt-model-evaluator.md index 592770f11f2..f9a46089e5e 100644 --- a/plugins/modelopt/agents/modelopt-model-evaluator.md +++ b/plugins/modelopt/agents/modelopt-model-evaluator.md @@ -18,4 +18,6 @@ Before acting, load these Model Optimizer instructions: Use matched baseline and candidate configurations. Complete the NEL dry-run, canary, full-run, and completed-run validation gates. Configure and verify MLflow export. Never report scores from an incomplete or invalid run. +Apply `evaluation/references/mlflow-verification.md` independently of evaluation acceptance. Recover export from existing results only; never submit another evaluation to repair delivery. In `MLflow`, report each task's export outcome and verified run URL (or none verified), mapped to its invocation ID. Keep evaluation and export outcomes separate; report blockers and preserved evidence paths when delivery fails. + Return only a concise handoff with these headings: `Status`, `Evaluation role`, `Checkpoint`, `Configuration`, `Results`, `Validation`, `MLflow`, `Artifacts`, and `Blockers`. Include invocation IDs, task-to-score mappings, score fields, sample accounting, and absolute paths. Do not return raw logs. diff --git a/plugins/modelopt/skills/evaluation/SKILL.md b/plugins/modelopt/skills/evaluation/SKILL.md index da57288528f..4452fbbbfd6 100644 --- a/plugins/modelopt/skills/evaluation/SKILL.md +++ b/plugins/modelopt/skills/evaluation/SKILL.md @@ -43,7 +43,7 @@ for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead: 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). 3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block. -4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r -c ` after (config-driven; see `references/nel-next.md`). +4. Follow `references/nel-next.md`'s run flow: dry-run → canary → full → checked MLflow push → delivery verification. Steps 1–9 below are currently validated with 0.2.6 — use them for everything else. @@ -155,7 +155,7 @@ Ask the 5 questions via AskUserQuestion (categories must match `nel skills build 1. **Execution:** Local / SLURM 2. **Deployment:** None (External) / vLLM / SGLang / NIM / TRT-LLM. Prefer vLLM unless the user/card says otherwise. -3. **Auto-export:** None / MLflow / wandb +3. **Auto-export:** None / MLflow / wandb. 4. **Model type:** Base / Chat / Reasoning 5. **Benchmarks** (multi-select): standard / code / math_reasoning / safety / multilingual @@ -501,6 +501,10 @@ Add credentials per the common skill's `slurm-setup.md` §6 if missing. If you c Run directly when the user asked to launch; otherwise ask before submitting. +**Before submitting any canary or full run** (including shortcut and existing +configs), apply `references/mlflow-verification.md#before-upload`: disable +auto-export if generated artifacts cannot be checked before automatic upload. + **Env setup:** `.env` is normally already created and filled back in Step 1 (via `modelopttools:eval-config`), at the **workspace root** — the dir you run `nel` from, not under the skill dir. Ensure it exists and source it — do **not** clobber an existing `.env`: ```bash @@ -558,10 +562,14 @@ Remove `limit_samples` overrides; keep canary-validated parallelism. If the cana --- -### Step 9 — Verify completed run +### Step 9 — Verify completed run and MLflow delivery Before pulling/reporting scores, validate the run. Read `references/run-validation.md` for NEL timeout/resume behavior, completed-run validation, diagnostics, and score harvesting. For a baseline that will be compared with a candidate, also perform its **External Baseline Sanity Check** before a success verdict, then hand the validated runs to `compare-results` for baseline-vs-candidate deltas. +Then apply `references/mlflow-verification.md`: verify each task's actual MLflow +run, recover failed/incomplete exports from existing results without rerunning +evaluation, and report evaluation and export outcomes separately. + --- Issues: · diff --git a/plugins/modelopt/skills/evaluation/references/mlflow-verification.md b/plugins/modelopt/skills/evaluation/references/mlflow-verification.md new file mode 100644 index 00000000000..ea57ff7da96 --- /dev/null +++ b/plugins/modelopt/skills/evaluation/references/mlflow-verification.md @@ -0,0 +1,91 @@ +# Verify MLflow delivery + +Evaluation completion and MLflow delivery are separate outcomes. Apply this gate +per invocation/task after run validation, before handoff or cleanup. Use +`launching-evals` to locate existing results and `accessing-mlflow` to query the +tracking server; those vendored skills remain unchanged. + +## Before upload + +Before any upload (automatic or manual), secret-scan all outgoing artifacts, +including result bundles, configs and logs; redact credentials, tokens, +authorization headers, and secret-bearing URLs. Never upload `.env`/secret files. +Preserve non-secret methodology/provenance in sanitized copies; if safe upload +cannot be established, report export blocked. + +Before submission, if generated artifacts cannot be checked before automatic +upload, disable auto-export (`execution.auto_export.destinations: []` for NEL +0.2.6) for both canary and full runs. Keep the MLflow export settings for checked, +sanitized manual delivery from existing results below. A post-run scan cannot +satisfy this prerequisite: 0.2.6 submits its exporter without an agent-review pause. + +## Verify before exporting again + +1. Record the invocation ID, task/job identity, evaluation outcome, and result + paths. Check the separate auto-export job's state and logs: evaluation + `SUCCESS` does not prove export succeeded. If export is queued/running, monitor + it or report export pending; do not start a competing export. +2. Search the configured MLflow server/experiment by `invocation_id` and task + identity, not display name alone. Search other experiments if necessary. + Distinguish canary/full runs and repeated tasks. For nel-next, use its run ID + and bundle `job_id` identity instead. An inaccessible server is **unverified**, + not evidence that no run exists. +3. Inspect each actual run's metrics and artifact contents, not just its status + or artifact listing. Verify the canonical score field/value against the + validated results, the evaluated checkpoint and resolved configuration + (including task overrides), and sample/scoring coverage. Check required + diagnostics: result file, resolved config, runtime metrics when produced, + and relevant client/server/SLURM/judge logs or compact diagnostic evidence + supporting the validation summary. Check benchmark-specific artifacts where + required. Record genuinely inapplicable diagnostics; missing required + evidence means incomplete export. Artifact paths vary: discover them. +4. Reuse a usable run. Return its verified run URL and invocation/task mapping; + never infer a run URL from an invocation ID or accept a same-named run. + +## Recover existing results only + +If auto-export failed or the run is incomplete, first confirm no exporter is +still active and recheck MLflow immediately before recovery. Inspect the +**installed** launcher's version and help (e.g. `nel --version`, `nel --help`, +then `nel export --help` if that subcommand exists). Use only the destination, +invocation/task selection, config, and artifact/log options that version supports. + +Manually export the **existing invocation/results** to MLflow. Do not run or +resume evaluation to repair export. Diagnose the export-specific failure: for +example, a CPU export job's container-import failure needs an available, +compatible export image or supported local export, not another GPU evaluation. +Change only export settings using the installed version's supported mechanism; +retain the original evaluation configuration and record recovery settings +separately. A local summary export is not MLflow delivery or an artifact backup. + +If a partial run exists, prefer a supported update/repair of that run. Do not +assume retries are idempotent: establish the exporter's behavior first. If a +replacement is unavoidable, record both run IDs and identify the verified +replacement; do not delete the original without authorization. After recovery, +repeat the full verification above. Export exit code zero alone is insufficient. + +**nel-next:** Preserve the explicit `nel-next.sh mlflow-push -r -c ` +workflow in `nel-next.md`; SLURM does not auto-export there. The wrapper stages +only `eval-*.json`, not logs; `copy_logs=true` cannot supply unstaged evidence. +Verify the pushed runs, then attach missing sanitized configs/logs or compact +diagnostic evidence to the identified runs using a supported MLflow artifact +upload. Recheck their contents; if evidence is unavailable, report incomplete or +blocked delivery. Do not substitute the legacy launcher's export command. + +## Safe evidence and blocked delivery + +If recovery remains blocked, preserve a **small, sanitized** evidence bundle in +an approved durable location before mandated cleanup: invocation/task and job +IDs, scores/results, evaluated config, coverage and diagnostic summary, relevant +log excerpts, export error, launcher version, and attempted recovery command +(with secrets removed). Include any known partial MLflow run IDs and missing +artifacts. Do not copy caches, checkpoints, or all traces as an export fallback. +If evidence cannot be preserved, report that blocker before destructive cleanup; +never silently discard the only results. + +In the evaluator handoff, report evaluation and export outcomes independently +per task: e.g. `evaluation: valid; export: verified (auto/manual)`, `pending`, +`incomplete`, or `blocked`. Include invocation IDs, verified MLflow run URLs (or +explicitly none verified), missing evidence, the specific blocker, and absolute +preserved-evidence paths. A valid evaluation may have blocked delivery; neither +outcome overrides the other or authorizes another evaluation submission. diff --git a/plugins/modelopt/skills/evaluation/references/nel-next.md b/plugins/modelopt/skills/evaluation/references/nel-next.md index 8e009098d9f..596abc04176 100644 --- a/plugins/modelopt/skills/evaluation/references/nel-next.md +++ b/plugins/modelopt/skills/evaluation/references/nel-next.md @@ -199,7 +199,7 @@ with its own `run_id`, copying the shared `services:` block. containing `${` — `experiment_name` included, since the same pass resolves the block. Keep `tracking_uri` as `${MLFLOW_TRACKING_URI}` — cross-server, the inherited name makes a same-named experiment here and `modelopt_run_url` is the only route back. -## Run (dry-run → canary → full) → push to MLflow +## Run (dry-run → canary → full) → push → verify MLflow delivery ```bash set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh" @@ -207,13 +207,24 @@ set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh" "$NEL" eval run .yaml --submit -O benchmarks.0.max_problems=2 -O benchmarks.0.repeats=1 -O benchmarks.0.max_concurrent=2 # canary "$NEL" eval run .yaml --submit # full "$NEL" eval {status|logs -f|report -f markdown|merge} -r # lifecycle -"$NEL" mlflow-push -r -c .yaml # post-run: push merged bundle(s) to MLflow ``` +Before pushing, apply [Before upload](mlflow-verification.md#before-upload) to +all outgoing bundles and export metadata. The wrapper fetches the cluster's +`eval-*.json` unchanged; if they need redaction, use the installed nel-next +exporter's supported local-bundle path with sanitized copies instead. + +```bash +"$NEL" mlflow-push -r -c .yaml # post-run: push checked merged bundle(s) +``` + +Then apply [MLflow delivery verification](mlflow-verification.md), including its +nel-next evidence-repair path, before handoff or cleanup. + `eval run` on a slurm cluster scp's the sbatch + redacted `.secrets.env` and submits via SSH; a built-in afternotok chain auto-resumes across walltime windows; -sharded runs auto-merge. **SLURM does not auto-export** — `mlflow-push` is the final -step: it reads the config's `export_config.mlflow`, stages each merged bundle's +sharded runs auto-merge. **SLURM does not auto-export** — `mlflow-push` reads the +config's `export_config.mlflow`, stages each merged bundle's `eval-*.json` off the cluster (the dev box doesn't mount the run dir), and exports with `emit_traces=false` (the default emits one trace per sample → minutes-long hang). Idempotent (re-push updates the same run, deduped by `job_id`); forward extra exporter diff --git a/plugins/modelopt/skills/evaluation/tests/evals.json b/plugins/modelopt/skills/evaluation/tests/evals.json index 3a33a6af053..c7bedcdc004 100644 --- a/plugins/modelopt/skills/evaluation/tests/evals.json +++ b/plugins/modelopt/skills/evaluation/tests/evals.json @@ -72,12 +72,76 @@ "Provides SSH-based log monitoring commands for SLURM execution" ] }, + { + "name": "mlflow-pre-submit-safeguard", + "skills": ["evaluation"], + "query": "Launch using my existing NEL 0.2.6 config with MLflow auto-export enabled. Generated configs and logs cannot be checked before the CPU exporter uploads them.", + "files": [], + "expected_behavior": [ + "Applies the common Step 8 pre-submit safeguard despite skipping config generation; the same gate applies to shortcut-generated configs", + "Sets execution.auto_export.destinations to [] before both canary and full submission, retaining MLflow export settings", + "Plans checked, sanitized manual delivery from existing results rather than relying on a post-upload scan" + ] + }, + { + "name": "mlflow-auto-export-verification", + "skills": ["evaluation"], + "query": "My full evaluation succeeded. The CPU auto-export is still running and MLflow has a same-named canary run. Finish the handoff once export completes.", + "files": [], + "expected_behavior": [ + "Checks the separate export job and does not start a competing manual export while it is active", + "Finds each full-run MLflow run by invocation and task identity rather than accepting the same-named canary", + "Verifies canonical score, evaluated configuration, coverage, and required diagnostic artifact contents against existing results", + "Returns verified task-to-run URLs and invocation IDs with separate evaluation and export outcomes, without exporting again" + ] + }, + { + "name": "mlflow-manual-export-recovery", + "skills": ["evaluation"], + "query": "Evaluation completed, but the CPU export job failed importing python:3.12.7-slim. One task has a partial MLflow run missing config and diagnostics; another has no run. Recover their exports without rerunning evaluation.", + "files": [], + "expected_behavior": [ + "Checks export job state and existing runs by invocation/task identity, rechecking before recovery to avoid duplicates", + "Inspects the installed launcher version and CLI help before choosing supported export, image/config, and artifact options", + "Repairs or manually exports existing results only; never runs or resumes evaluation to repair export", + "Determines whether partial-run repair is supported instead of assuming export idempotency; records any unavoidable replacement run", + "Secret-scans and redacts uploaded configs/logs without removing non-secret evaluation provenance", + "Verifies scores, configs, and diagnostics in MLflow after recovery; returns actual URLs and invocation IDs with export marked manually verified" + ] + }, + { + "name": "mlflow-export-blocked-before-cleanup", + "skills": ["evaluation"], + "query": "Four tasks completed, but all auto-exports failed. Manual export is also blocked by MLflow authentication. Cleanup of the cluster results is due. Report the outcome.", + "files": [], + "expected_behavior": [ + "Reports evaluation outcomes separately from blocked or unverified MLflow delivery for all four tasks", + "Names the authentication blocker and does not fabricate URLs, infer absent runs from access failure, or claim export success", + "Preserves small sanitized results/config/coverage/diagnostic and export-failure evidence with invocation/task IDs in an approved durable location before cleanup", + "Reports absolute evidence paths, launcher version and sanitized recovery attempt; flags inability to preserve evidence before destructive cleanup", + "Does not upload secrets, copy bulk caches as backup, or rerun evaluation" + ] + }, + { + "name": "nel-next-explicit-push-verification", + "skills": ["evaluation"], + "query": "My nel-next SLURM evaluation completed. The merged bundles are safe to upload, but the wrapper stages only eval-*.json; logs remain on the cluster. Deliver verified MLflow results.", + "files": [], + "expected_behavior": [ + "Preserves the explicit nel-next.sh mlflow-push -r -c workflow rather than expecting SLURM auto-export or substituting legacy export syntax", + "Checks outgoing bundles and export metadata before pushing and follows the verification gate directly from the nel-next run flow, without relying on excluded Steps 1-9", + "Verifies each pushed run by run/bundle job identity, scores, configuration, and required diagnostics before returning actual URLs", + "Attaches missing sanitized diagnostic evidence to the identified runs and rechecks contents; does not assume copy_logs=true uploads unstaged logs", + "Reports incomplete or blocked delivery if required evidence cannot be delivered, without rerunning evaluation" + ] + }, { "name": "scicode-eight-run-average", "skills": ["evaluation"], "query": "Evaluate my NVFP4 checkpoint on the AA suite including SciCode on SLURM", "files": [], "expected_behavior": [ + "Applies Step 8's pre-upload safeguard on the AA shortcut path, disabling auto-export before canary and full submission if generated artifacts cannot be checked before upload", "Keeps num_repeats at 1 and plans at least 8 independent SciCode submissions instead", "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite", "Reports the SciCode score as the mean of the pooled runs with standard error and run count",