perf: take Langfuse trace linking off the eval item critical path

.gitignore

@@ -42,3 +42,10 @@ packages/gooddata-sdk/tests/export/exports/default/
 AGENTS.md
 .aiassistant/rules/aida.md
 .junie/guidelines.md
+
+# gooddata-eval local run artifacts. Root-anchored on purpose: a bare `datasets/` would
+# also shadow packages/gooddata-pandas/tests/.../ldm/datasets/, which is tracked.
+/packages/gooddata-eval/EVAL_results
+/packages/gooddata-eval/datasets/
+# MCP tool logs, written to a relative path by whatever is started from the repo root
+/logs/

packages/gooddata-eval/README.md

@@ -114,6 +114,7 @@ gd-eval run \
 |---|---|
 | `--dataset PATH` | Flat folder of JSON files — one question per file. |
 | `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--kind TEST_KIND` | Fallback `test_kind` for dataset items that do not embed one. Defaults to `visualization`; use e.g. `agentic_metric_skill` for multi-turn agentic evaluation. Items that declare their own `test_kind` ignore this. |
 
 #### Model selection
 
@@ -126,21 +127,62 @@ gd-eval run \
 | Flag | Default | Description |
 |---|---|---|
 | `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
-| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests. Progress output interleaves when K > 1. |
+| `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
+| `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
 
+**Concurrency and workspace safety.** Agentic kinds that create workspace objects
+(`agentic_metric_skill`, `agentic_alert_skill`, `agentic_conversation`, `agentic_kda_skill`) always run one at a
+time whatever `--concurrency` says — a metric or alert created and dropped mid-run would otherwise be visible to
+another item reading the same catalog. **That protection is for the agentic kinds only:** the single-turn
+`metric_skill` and `alert_skill` kinds are still fanned out and the agent performs the same server-side writes on
+that path, so avoid raising `--concurrency` on a dataset of those against a shared workspace. Progress output
+interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.
+
 #### Output
 
 | Flag | Description |
 |---|---|
 | `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
 | `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
+| `--preserve-failed` | Keep failed conversations on the server instead of deleting them, so they can be inspected afterwards. Applies to the single-turn chat path; agentic kinds manage their own conversation lifecycle. |
+| `--timers` | Print per-turn `[timer]` diagnostics — GoodData response, judge, and simulated-user seconds as they happen. Off by default: an 18-item `--runs 2` run emits ~72 lines and buries the progress output. The same measurements are always in the JSON report's `latency_breakdown_s`, so this only adds a live view. Also settable via `GD_EVAL_TIMERS=1`. |
 
 #### Langfuse sink
 
 | Flag | Description |
 |---|---|
-| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`, suffixed `-effort-{level}` when `--reasoning-effort` is set so runs differing only by effort stay separate). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Names each experiment run `{dataset_name}_{timestamp}_{model}`, suffixed `_effort-{level}` when `--reasoning-effort` is set (so runs differing only by effort stay separate) and `_run{N}` per run when `--runs` > 1 — e.g. `general_question_2026-09-02-11-13_gpt-5.2_run0`. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
+
+Set `TAVERN_E2E_SKIP_TRACE_LINK=1` to skip trace lookup entirely (scores are then orphaned; the run says so).
+
+**A local `--dataset` cannot be attached to a Langfuse run.** `--langfuse` is refused alongside `--dataset`
+because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that
+flag — each `evaluate_agentic_*` builds its own client whenever `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are
+exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping
+fails, with one `404 from dataset-run-items` reported per run. The run warns about this before it starts. Use
+`--langfuse-dataset` when you want runs that are comparable across models, or `TAVERN_E2E_SKIP_TRACE_LINK=1` to
+skip linking altogether.
+
+**When trace linking happens.** Finding a gen-ai trace means polling until Langfuse has ingested it, which is
+lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it
+does not run inline per item. Every item's Langfuse block is queued and the whole batch runs *after* the agent
+phase, draining before any report is written. Two consequences worth knowing:
+
+- **No item's `latency_s` includes trace linking.** Its cost is reported separately as
+  `latency_breakdown_s.langfuse_s`, and the run prints
+  `[langfuse] trace linking finished in Xs for N item(s); slowest Ys`. If `slowest` approaches the **120s**
+  batched retry budget, links are timing out and scores are being orphaned — look for
+  `[langfuse] WARNING: no trace found for conversation ...`.
+- **The budget depends on who is waiting.** 120s is affordable only because the batch blocks nobody. A direct
+  library caller (`evaluate_agentic_*` without a `submit_trace_link`) polls inline, on its own critical path, and
+  gets **35s** instead — the same cost as before batching existed, so no inline caller pays for a budget raised
+  on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps.
+Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops
+whatever is still queued rather than making you wait it out: both the queued trace links and, under
+`--concurrency`, the items that have not started. The handful of items already in flight still have to finish —
+worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one
+`--concurrency`-wide wave, not the rest of the dataset.
 
 ### JSON report shape
 
@@ -162,6 +204,70 @@ The JSON report always uses the nested multi-model shape:
 
 Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
 
+Each item reports **how many of its runs passed**, not only whether one did:
+
+```json
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+```
+
+`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
+`quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
+run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
+inconsistent rather than wrong. The console shows `4/5 runs passed` in `Notes` for a non-unanimous pass and
+stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on every run`.
+
+`runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
+no K and drives its fixture exactly once.
+
+Each item additionally carries a per-phase breakdown:
+
+```json
+"latency_breakdown_s": {
+  "agent_s": 4.02,          // GoodData's own response time — the system under test
+  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
+  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
+  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
+}
+```
+
+Every item reports `runs_ungraded` beside `runs_passed`: runs the agent answered but the LLM judge returned
+nothing readable for. Agentic items also list them as `unscored_runs` / `judge_errors` in their `detail`, and a
+`dashboard_summary` item carries `ungraded_criteria`. Such a run — or, for `dashboard_summary`, such a
+criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0
+would be indistinguishable from the judge genuinely failing the answer, which is the confusion
+`JudgeResponseError` exists to end. `pass@K` still holds on the runs that *were* graded, so an item can pass with
+`runs_ungraded` set; `pass^K` cannot, because a run nobody graded leaves "all K passed" unverified. For
+`dashboard_summary` an ungraded `must_include` / `must_not_include` criterion likewise cannot carry a pass —
+"the judge could not tell" is not evidence the fact is present — while an ungraded `rubric` line only narrows
+the quality score. When *no* run could be graded (for `dashboard_summary`: no gating criterion on any run) the
+item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than
+`--runs` asked for, so treat the result as weaker evidence and check the judge (`GD_EVAL_JUDGE_DIAGNOSTICS=1`,
+or raise `JUDGE_MAX_COMPLETION_TOKENS` if the cause is `finish_reason=length`). The console says so in `Notes`
+(`1 run(s) ungraded`, `2 criterion(s) ungraded`).
+
+`agent_s` + `judge_s` + `simulated_user_s` are the instrumented parts of the item's `latency_s`; they do not add
+up to it exactly, because `latency_s` is wall-clock around the whole item and also covers the conversation
+create/delete round trips, SDK construction and any cleanup. `langfuse_s` sits **beside** `latency_s`, never
+inside it, because trace linking runs outside every item's critical path (see above) — summing all four would
+re-inflate exactly what that design removes.
+
+A phase that a kind does not have reports `0.0` rather than an invented number, so read the zeroes as "not
+applicable here", not "instant". Today:
+
+| Field | Populated by |
+|---|---|
+| `agent_s` | `agentic_general_question`, `agentic_metric_skill` |
+| `judge_s` | `agentic_general_question` only — `agentic_metric_skill` compares MAQL by string, it has no LLM judge |
+| `simulated_user_s` | `agentic_metric_skill` only — `agentic_general_question` is single-turn, it has no simulated user |
+| `langfuse_s` | every agentic kind, but only on the `gd-eval` path and only when Langfuse credentials are present |
+
+The other six agentic kinds report `0.0` for the first three. Trace linking itself happens whenever
+`LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY` are exported, with or without `--langfuse`, because each
+`evaluate_agentic_*` falls back to `try_make_langfuse_client()`. But its *duration* is measured by the CLI
+runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
+linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
+
 ---
 
 ## `gd-eval models`

packages/gooddata-eval/pyproject.toml

@@ -30,7 +30,7 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
-llm-judge = ["openai>=1.40,<2.0"]
+llm-judge = ["openai>=1.45,<2.0"]
 
 [project.scripts]
 gd-eval = "gooddata_eval.cli.main:main"