fix(errors): classify agent-session/Exec-format failures + 3-way transient/service/user axis with auto-retry-once#599
fix(errors): classify agent-session/Exec-format failures + 3-way transient/service/user axis with auto-retry-once#599isadeks wants to merge 11 commits into
Conversation
…apper The error classifier keyed on `agent_status=error_max_turns` (pipeline.py's wrapper) but missed runner.py:515's `Agent session error (subtype='error_max_turns')`. A real max-turns failure fell through to UNKNOWN → 'Unexpected error' (live-caught ABCA-483: a task hit the 100-turn cap but the Linear reply read 'Unexpected error'). The max_turns / max_budget / error_during_execution patterns now match BOTH the `agent_status=` and `subtype=` wrappers. Platform-agnostic, MAIN-BOUND (cherry-pick clean: single file + colocated test, no Linear/orchestration coupling).
…y-once on transient session-start Error handling couldn't distinguish a transient hiccup from a real service error — the retryable boolean conflated 'self-heals on retry' with 'you must change something first', so a customer couldn't tell whether to retry or call their admin. - error-classifier.ts: new ErrorClass axis (transient / service / user) on every classification. transient = infra/service hiccup a retry clears (ECS deploy-race, ENI/capacity delay, network blip, Bedrock throttle, concurrency cap); service = real platform/config fault an admin owns (bad token/scopes, model-not-enabled, blueprint misconfig); user = the request/code is the thing to change (build/test failed, guardrail, wrong PR, max-turns). retryGuidance() keys on errorClass and takes an autoRetried flag. New isTransientError() helper. - orchestrate-task.ts: AUTO-RETRY-ONCE at session-start for transient failures — the one place a retry is idempotent by construction (no clone/commits/PR yet). Emits session_start_retry; stamps '[auto-retried]' so the surface says 'I tried again, still failed'. Mid-run crashes are NOT retried. Backport note: the failure-reply.ts consumer hunk from the origin commit is omitted — that module is part of the not-yet-merged #247 orchestration UX and does not exist on main. The classifier axis + auto-retry stand alone. (cherry picked from commit 63a12dc)
… retry guidance (ABCA-659) When the agent image's claude CLI can't be exec'd (OSError Exec format error, or the shim's own 'claude native binary not installed'), the classifier had no matching pattern and fell through to the bare 'Unexpected error' with a generic 'report it' remedy — the exact no-guidance anti-pattern the error-feedback arc (K5 buckets, remedy+retryable+errorClass) was built to eliminate. Live-caught on ABCA-659's retry: the ❌ Linear comment just said 'Unexpected error', telling the user nothing about whether to retry or escalate. Add a COMPUTE + transient pattern so the failure reads honestly and actionably: 'Couldn't start the coding agent (environment issue) … not a problem with your request … reply here to try again; if every attempt fails the same way, the agent image needs a rebuild — contact your ABCA admin with the task id.' The transient class also lets the platform's session-start auto-retry take a shot. Matched before the AGENT/UNKNOWN fallthrough; two tests pin the copy + axis.
7128618 to
77042c8
Compare
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (Stack B: agent error handling, PR 1/2)
Base verified against fresh origin/main. Stacked: #600 builds on this branch. All mandatory agents run (table below). Dead-code ratchet status verified by actually running knip.
Verdict: Request changes
Strong error-classifier work — the subtype= wrapper fix (ABCA-483: real max-turns failures no longer read as "Unexpected error"), the transient/service/user ErrorClass axis with fail-safe defaults (absent ⇒ USER, never auto-retry an unknown), and the TaskDefinition is inactive / claude Exec format error classifications. Pattern ordering verified correct (specific patterns precede the catch-alls). Two issues hold it: one MEDIUM correctness bug and a crit-9 test gap on the riskiest new behavior.
Blocking issues
B1 — MEDIUM: the unguarded session_start_retry emit can falsely stamp [auto-retried] and replace the real failure cause. orchestrate-task.ts:189-197:
autoRetried = true; // set first
await emitTaskEvent(taskId, 'session_start_retry', {...}); // UNGUARDED
handle = await strategy.startSession(startInput); // retry — never runs if the emit threwemitTaskEvent (orchestrator.ts:205) is a bare await ddb.send(new PutCommand(...)) with no internal try/catch — it throws on any DynamoDB fault. If the TaskEventsTable PutItem throttles/times out (exactly the conditions co-occurring with the transient session-start failures this path handles), it throws after autoRetried=true but before the retry executes. The outer catch then produces Session start failed: <DDB error> [auto-retried]: the user is told a second attempt was made and failed when none ran, and the real retriable cause is discarded for an observability-layer error they can't act on. This breaks the file's own best-effort-emit convention (the concurrency-cap path at :89 deliberately guards its emits).
- Fix: wrap the emit in try/catch (WARN-and-continue) so a telemetry failure never aborts or mis-attributes the retry. Also pass the
correlationenvelope — it's dropped here vs. the siblingsession_startedemit (minor #245 trace gap; fold into the same fix).
B2 — crit-9 test gap: the session-start auto-retry has ZERO coverage. orchestrate-task.test.ts has no test for retry / session_start_retry / [auto-retried]. The retry lives inline in the durable handler's start-session step, which the suite never invokes (it tests exported shared/orchestrator helpers in isolation). All four branches are unverified — most importantly "non-transient failure must NOT retry" (where a mis-classification wastes a retry or doubles a non-idempotent op) and the transient→retry→succeeds happy path the feature exists for.
- Fix: extract the retry into an exported
startSessionWithRetry(strategy, input)(matches the existing "test the exported helper" pattern) and unit-test the 4 branches; or add a durable-handler test with a mock strategy that rejects-then-resolves.
Non-blocking suggestions / nits
- N1 (LOW — confirmed by two agents) —
retryGuidance(),AUTO_RETRIED_MARKER, andrenderFailureReplyhave no production consumer onmain(the latter two don't exist in the repo;retryGuidanceis called only by its own test). Theorchestrate-task.ts:226comment forward-referencesrenderFailureReply/AUTO_RETRIED_MARKERas if they exist. Not a runtime bug (the[auto-retried]string is persisted verbatim and doesn't affect classification), but the "I already tried again" copy the feature advertises is never rendered. Either wireretryGuidanceinto the failure renderer, or soften the comments to "will be consumed by the forthcomingrenderFailureReply" and noteretryGuidanceships ahead of its consumer. - N2 (LOW) — The retry classifies the re-wrapped string
`Session start failed: ${firstErr}`, and since/Session start failed/iis itself TRANSIENT and sits after only a few specific patterns, a genuinely non-transient fault (missing-ECS-env config error, rawAccessDenied) classifies TRANSIENT and eats one ~1-min retry. Safe (never masked as success), but the inline comment's claim "a NON-transient failure throws immediately" is only true for faults matching an earlier SERVICE pattern. ClassifyString(firstErr)directly, or soften the comment.
Tests & CI
- CI: all green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions,
build (agentcore));MERGEABLE. - Dead-code ratchet — ran it for ground truth. It reports count 81 > baseline 78 on this head, but it's advisory-only, not blocking: not part of
mise run build, and the CI job iscontinue-on-error(mise.toml: "Advisory in CI for now; flips to blocking once the baseline is driven to zero");origin/mainitself already sits at 80 > 78. The +1 knip attributes here isErrorClassType, which is used (error-classifier.ts:82, theerrorClass?field type) — a knip false-positive on a(typeof X)[keyof typeof X]alias, not real dead code. No action required. - Coverage: classifier axes are genuinely well-tested (the every-classification-has-errorClass invariant loop is high-value). The gap is the auto-retry → B2.
- Bootstrap synth-coverage: N/A (classifier + handler logic; no CDK construct/stack/resource change).
Review agents run (Stage 3 — mandatory)
| Agent | Ran? | Outcome |
|---|---|---|
code-reviewer |
✅ | No blocking logic defects; flagged the over-broad-but-safe retry gate (N2) + unwired retryGuidance (N1). |
silent-failure-hunter |
✅ | 1 MEDIUM → B1 (unguarded retry-event emit). Confirmed the classifier's fail-safe defaults (absent ⇒ USER, isTransientError false for absent) are correct, non-masking. |
pr-test-analyzer |
✅ | crit-9 → B2 (auto-retry untested); classifier axes well-covered. |
type-design-analyzer |
⏭️ omitted | ErrorClass/ErrorClassType are simple const-object enums; no complex new type. |
comment-analyzer |
⏭️ omitted | Comment accuracy covered by hand + code-reviewer/silent-failure (N1/N2 are the comment findings). |
/security-review |
⏭️ omitted | No IAM/Cedar/network/secrets/input-gateway change. |
Human heuristics
- Proportionality — ✅ Matches documented live incidents (ABCA-483/659); minimal surface.
- Coherence — ✅
ErrorClassunifies retry semantics repo-wide; #600 stacks cleanly on it. - Clarity —
⚠️ Excellent "why" comments, but two forward-reference code that doesn't exist yet (N1) or overstate a guarantee (N2). - Appropriateness — ✅ Sound; the one gap (B1) is that the retry-event emit should follow the file's own best-effort pattern.
Governance
Branch pr/error-classifier-hardening carries no issue number and the body references only linear-vercel — no approved GitHub issue (ADR-003). Please attach an approved issue + conforming branch before merge (a natural sibling of the ABCA-483/659 incidents). Not gating the code verdict, but required to merge.
Bottom line: fix B1 (guard the retry-event emit — a real MEDIUM that can misreport failures) and B2 (test the untested auto-retry, ideally by extracting it); N1/N2 are comment/wiring cleanups. The classifier work is high quality.
🤖 Generated with Claude Code
…urns cap (ABCA-662) (#600) * feat(agent): stuck/runaway guard — break a repeating-failing-command loop Live-caught ABCA-483: a one-line README task burned ALL 100 turns (~22 min, $1.53) because the agent re-ran the SAME failing command (mise //cdk:test → JS-heap OOM, exit 134) over and over, yak-shaving the build env instead of finishing. Nothing noticed until the hard max_turns cap killed it. New `stuck_guard.py` (pure, dep-free): PostToolUse records each tool result; a coarse (tool_name, command) signature tracks consecutive failures. A new between-turns hook (registered after cancel, before nudge — same short-circuit rationale) then: - STEER once at 3 repeats: inject 'stop retrying X — work around it or finish with what you have'; - BAIL at 6 repeats: end the turn loop early (continue_=False) so the task fails fast instead of grinding to the cap. Conservative by design: keys on a REPEATING FAILURE (not a raw turn count), so a large task making varied progress never trips it; unknown tool output counts as success. Fail-open everywhere — a guard bug can't wedge the agent. Platform-agnostic, MAIN-BOUND: lives entirely in agent/src (stuck_guard.py + hooks.py); surfaces via the channel-neutral error_message. The honest reason is then rendered per-channel by the classifier / failure-reply (several fixes). 1158 agent tests pass (ruff + ty clean). * fix: stuck-guard is advisory-only — drop the bail, keep a one-time steer Reviewing the guard for false-positive risk (user: 'make sure the turn limit isn't too aggressive') showed the bail path was unsafe: distinguishing a true spin from a legitimately-iterating agent (re-running the same test command as it fixes failures one by one) from raw output is genuinely fragile — a digit- normalizing fingerprint that ignores volatile GC timings ALSO collapses 'test file_0' vs 'file_1' (the progress signal), so it would kill a working agent. A false-positive KILL is far worse than a false-positive nudge. So: removed BAIL entirely. The guard now only ever injects a ONE-TIME advisory steer when the same command fails with IDENTICAL output N times; the max_turns cap (with a fix's honest 'Exceeded max turns' reason) is the real runaway backstop. A false positive costs exactly one extra advisory comment. Platform-agnostic, main-bound. * fix(agent): explain WHY a task hit max_turns + catch loop-of-variations early (ABCA-662) ABCA-662 capped at max_turns, but the CAUSE was masked: it had finished the code and then thrashed ~15 turns retrying a failing 'git push' (remote: invalid credentials) every which way — a DIFFERENT command each turn, so the existing per-signature stuck-guard (same command × identical output ≥3) never tripped, and the terminal reason read only 'Exceeded max turns'. max_turns is a CORRECT classification, but the user can't tell 'genuinely needed more turns' from 'spun on an error' — and raising the cap just burns more tokens. Two changes, both advisory (no hard kill — keeps the K10 no-false-kill stance): - stuck_guard: add a signature-agnostic TRAILING WINDOW (last 6 tool outcomes). When ≥5 share the SAME byte-identical failure fingerprint across DIFFERENT commands, steer once ('you're spinning on one failing op — stop, it won't resolve by retrying'). EXACT fingerprint (no digit-blur) so a healthy iterate-and-fix loop (same cmd, a different test failing each run) never trips it — the K10 case, still covered by its test. Also exposes recent_failure_ summary(). - pipeline/hooks: the between-turns hook latches that summary; the terminal path appends it to a max_turns reason → 'error_max_turns … — spinning on failing tool calls (last: git push → invalid credentials)'. Only enriches max_turns; a productive run adds nothing. - error-classifier: a new bucket for 'error_max_turns … spinning on failing tool calls' → 'Ran out of turns while stuck on a failing step' with the honest remedy (raising --max-turns won't help; fix the failing op / check the env), ordered before the generic max_turns bucket. Tests: window-spin + K10 no-trip + summary + pipeline enrichment + classifier bucket. * fix(errors): don't over-claim on a max_turns spin — offer BOTH remedies (ABCA-662 review) The prior 'spinning on failing tool calls' classifier bucket asserted 'raising --max-turns won't help'. That over-claims: the trailing window can't distinguish (a) a HARD blocker no turn count fixes from (b) a LONG task that hit a transient/recoverable snag near the end and just ran out — which is 662 itself (siblings 661/663 pushed fine with the same token → its 'invalid credentials' was a transient race that more turns / a retry WOULD have cleared). Reframe the bucket to SURFACE what it was stuck on and present BOTH paths (environment blocker → fix + retry; transient/recoverable or a shorter sibling got past it → just retry / raise --max-turns), letting the failure KIND in the detail tell the reader which applies. Title 'Ran out of turns retrying a failing step'. The early window-steer (agent told to STOP and report while it still has turns) is the real mechanism that lets a long task escape the thrash; this copy is just the honest post-hoc explanation. Test updated to assert both remedies, not the over-claim. * fix(errors): max_turns stays "Exceeded max turns" — observe the repeated failure, don't diagnose it Reviewer concern on the ABCA-662 work: re-titling a max_turns failure as "Ran out of turns retrying a failing step" over-claims. The stuck-guard's trailing window is only the last ~6 tool calls, which genuinely CANNOT distinguish a hard blocker (bad creds, no permission — more turns won't help) from a long task that made real progress and hit a recoverable snag only at the tail (more turns / a retry WOULD help — 662 itself: siblings pushed fine with the same token). Framing the whole run around its last 6 calls misrepresents the latter. - error-classifier: drop the separate "retrying a failing step" bucket. A max_turns failure stays the plain "Exceeded max turns"; the description/remedy point the reader at the observed detail and still surface the environment-blocker path, but assert NO cause. - stuck_guard.recent_failure_summary: emit a NEUTRAL observation ("last tool calls repeated: `<cmd>` → <err>") instead of "spinning on failing tool calls". The mid-run steer TO the agent (an advisory nudge, not a user surface) still says "spinning" — that's fine, it's coaching the agent, not classifying the outcome. - tests updated to pin the neutral wording + assert the title is NOT re-framed.
… raw error Addresses the #599 review (Stack B): - B1 (MEDIUM): the `session_start_retry` telemetry emit was unguarded — a TaskEvents PutItem fault (throttle/timeout, co-occurring with the transient session-start failures this path handles) threw AFTER `autoRetried=true` but BEFORE the retry ran, so the user was told a second attempt failed when none had, and the real retriable cause was discarded. The emit is now best-effort (try/catch → WARN-and-continue) so telemetry never aborts/mis-attributes the retry. The `correlation` envelope is threaded so the retry event carries the same trace context as `session_started` (#245). - B2 (crit-9): the auto-retry had ZERO coverage (it lived inline in the durable handler's start-session step the suite never invokes). Extracted to an exported `startSessionWithRetry(strategy, input, deps)` — matching the repo's "test the exported helper" pattern — and unit-tested all four branches (success / non-transient no-retry / transient→retry→success / transient→transient→throw) plus the B1 best-effort-emit guard. - N2: classify the RAW error, not a ``Session start failed: …`` wrapper. The wrapper string itself matched a TRANSIENT pattern, so EVERY session-start failure classified transient and the "non-transient throws immediately" branch was effectively dead (a config/auth fault ate a pointless ~1-min retry). Classifying the raw string restores that branch (verified: TaskDefinition- inactive → retry; ECS_CLUSTER_ARN-not-configured / AccessDenied → surface). - N1: soften the `[auto-retried]` comment — it forward-referenced renderFailureReply/AUTO_RETRIED_MARKER that don't exist yet. Now states the marker is persisted verbatim for a forthcoming failure renderer (retryGuidance ships ahead of its consumer); no behaviour claim about rendering today. Full cdk build green (compile + jest + eslint + synth); 5 new helper tests pass.
|
Thanks — addressed both blockers + both nits in B1 (unguarded retry emit) — extracted the retry into B2 (untested auto-retry) — took the extract-and-test path: N2 (over-broad retry gate) — this was sharper than LOW: because the code classified the re-wrapped N1 (forward-ref comments) — softened the Dead-code ratchet — noted, agreed it's a knip false-positive on B2/governance — will file an approved GitHub issue (sibling of ABCA-483/659) + rename the branch to Full cdk build green (compile + jest + eslint + synth). Re-requesting review. |
…ary (#600 N3/N4) Follow-up coverage from the #600 review (feature approved as-is; tests only): - N3 (crit-8 wiring seam): the max_turns reason-enrichment was tested only at its two pure endpoints (stuck_guard.recent_failure_summary + the classifier on a hand-built string) — the append in _resolve_overall_task_status that joins them was unverified. Add TestMaxTurnsStuckEnrichment (monkeypatching hooks.last_stuck_summary): appends on error_max_turns, does NOT append on a non-max_turns error, leaves the reason unchanged when there's no summary, and does not double-append when the summary is already present. - N4 (crit-6 boundary): the window-steer was tested at 6/6 (steers) and 4/6 (doesn't) but not at the exact WINDOW_FAIL_THRESHOLD `>=` edge. Add test_window_steers_at_exactly_the_threshold (5/6 same-fingerprint fails in a full window → steers) and test_no_steer_when_window_not_yet_full (5 fails in a length-5 history → no steer, since _dominant_window_failure requires a full window). Imports WINDOW/WINDOW_FAIL_THRESHOLD so the constants' boundary semantics are now pinned. Full agent suite green (1218); ruff format + check clean.
scottschreckengaust
left a comment
There was a problem hiding this comment.
Follow-up review — prior blockers addressed; B2 governance still stands
Re-reviewed the reworked PR (now +1442, new session-start-retry.ts + test_pipeline_outcomes.py since my CHANGES_REQUESTED). The code is in good shape and my two prior technical blockers are resolved; the remaining blocker is governance (B2).
✅ Resolved since my prior review
- B1 (retry logic untestable inline in the durable handler) — extracted to
cdk/src/handlers/shared/session-start-retry.tswith all four branches unit-tested (success / non-transient-throw / transient-then-success / transient-then-transient) plus a best-effort-emit-guard test. The emit is now wrapped so a TaskEvents PutItem fault can't abort or mis-attribute the retry. ✅ - N2 (classify the raw error, not the
Session start failed:wrapper) — fixed.startSessionWithRetryclassifiesString(firstErr); classifying the wrapper would have matched the/Session start failed/itransient pattern and made every session-start failure retry (turning the non-transient-throw branch into dead code). Good catch. ✅ - Verified independently: the stuck-guard is genuinely advisory-only (can only return
none/steer, never kills); cancel-wins is preserved (cancel index 0 short-circuits before stuck-guard/nudge); the fail-open swallows inhooks.py/stuck_guard.pydon't alter the screening decision or terminal-status resolution; the regex additions sit before the generic catch-all. No blocking code defects.
🔴 Blocking — B2: governance (ADR-003) — unchanged
Still no approved issue on a conforming branch:
- No linked closing issue; the PR is labeled (
bug/validation-loop/orchestration/P1) but that is not theapprovedgate. I searched — there's no open issue withapprovedcovering the error-classifier/stuck-guard work (the ABCA-483/659/662 references are Linear ids, not GitHub issues). - Branch
pr/error-classifier-hardeningdoesn't follow(feat|fix)/<issue-number>-short-description.
Per ADR-003: file (or link) an issue with acceptance criteria, get the approved label, self-assign, comment "Starting implementation," and re-cut on a conforming branch (e.g. fix/<n>-error-classifier-hardening). This is the merge gate; the code itself is mergeable-quality.
🟠 New non-blocking finding — CLI type drift
#599 adds errorClass?: ErrorClassType to ErrorClassification in cdk/src/handlers/shared/error-classifier.ts, which serializes into the GET /tasks/{id} response (types.ts:252 error_classification → :607 classifyError(...)). The mirror in cli/src/types.ts:72-78 was not updated — it still lacks errorClass?. Per the CLAUDE.md routing rule (shared API shapes must stay in sync), add:
readonly errorClass?: 'transient' | 'service' | 'user';Not a build-breaker (additive optional; extra JSON is ignored, CLI compiles), but it's real drift and the CLI's structured error display can't surface the new axis until it's mirrored.
🟡 Nits (non-blocking)
retryGuidance()+[auto-retried]are dead onmain(no consumer until the #247 failure-renderer lands). The PR is explicit that they "ship ahead of the consumer," and the dead-code ratchet is advisory here, so acceptable — but keep the producer/consumer contract pinned by test (retryGuidance(c, true)copy is already asserted) so it can't rot before #247.- Retry is narrower than the prose implies — the comment cites "ENI/capacity delay, Bedrock throttle" but only
TaskDefinition is inactiveclassifies TRANSIENT today; rawThrottlingException/TimeoutErrorfall through to UNKNOWN→USER and are not retried (safe, but the named cases are latent until those patterns are added). pipeline.pymax_turns enrichment doesfrom hooks import last_stuck_summaryand appends unguarded. It can't realistically raise (trivial global read) and the whole body is insiderun_task's crash handler, so terminal status is always written — but a one-line local try/except around just the append would ensure an enrichment bug can never degrade the (already-correct) terminal reason.orchestrate-task.tswiring seam untested — the extractedstartSessionWithRetryis well covered, but no test drives the durable handler to assert it's actually called and that[auto-retried]is stamped on the double-failure path. A refactor reverting to a directstartSessionwould pass every test. Also 3 hook-glue branches (stuck-hook cancel short-circuit,evaluate()fail-open, the real producer→latch→consumer chain for the stuck summary) are untested.
Tests & CI
Coverage is strong and behavior-first: the 4-branch retry matrix, the errorClass invariant ("every pattern carries one"), the ABCA-662 neutrality assertions (title NOT re-framed, remedy points at the detail), and the stuck-guard boundary trio (exact >=/full-window edges, K10 false-positive avoidance). CI green, MERGEABLE.
🤖 Reviewed with Claude Code
…-hardening # Conflicts: # cdk/src/handlers/orchestrate-task.ts
#599 added `errorClass?: ErrorClassType` to the CDK ErrorClassification, which serializes into the GET /tasks/{id} `error_classification` response — but the hand-maintained CLI mirror (cli/src/types.ts) wasn't updated, so the CLI's structured error display couldn't surface the transient/service/user axis (real drift per the CLAUDE.md shared-API-shape routing rule). Add the optional field to the CLI's ErrorClassification, inlined as a union literal rather than a named `ErrorClassType` export — the types-sync drift check (scripts/check-types-sync.ts) requires CDK to be the source of truth for any exported type name, and CDK exports ErrorClassType from error-classifier.ts, not shared/types.ts. Inlining keeps the field synced without a CLI-only export. Verified: `//:check:types-sync` OK (57 CLI exports validated), cli build green (600 tests, compile + eslint clean).
|
Resolved the conflict + addressed the new CLI-drift finding ( Conflict (from #503 landing on main — both PRs touch New finding — CLI type drift: added Nits: N3-guard (try/except around the max_turns append) + the wiring-seam tests you flagged (nit 4) I'd fold into the existing #608 follow-up rather than widen this PR — they're the same "assert the durable handler actually calls the extracted helper" class. Nits 1/2 (retryGuidance ships-ahead-of-consumer, retry narrower than prose) are documented-as-intended. LMK if you'd rather any of those land here. B2 governance: filing the approved governing issue + branch rename to |
|
@scottschreckengaust — re-requesting review. B2 is cleared: governing issue #610 is now Same branch-name note as #596: renaming closes the open PR, so I've left it and linked the |
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (Stack B: error-classifier + stuck guard, re-review)
Re-reviewed against fresh origin/main (merge-base = main tip 22705e8; fully current). This PR is stacked — it carries the merged #600 stuck-guard work to main alongside the classifier hardening. All mandatory agents re-run on head 4e38a45. This supersedes my earlier CHANGES_REQUESTED.
Verdict: Approve (with nits)
Both of my prior technical blockers are resolved and governance is now cleared, so I'm approving. Summary of what closed since my last pass:
- B1 (unguarded retry-event emit could falsely stamp
[auto-retried]/ discard the real cause) — fixed: the retry is extracted intosession-start-retry.ts::startSessionWithRetry, the emit is wrapped best-effort (WARN-and-continue), and a telemetry fault can no longer abort or mis-attribute the retry. Verified bysession-start-retry.test.ts:94. - B2 (untested auto-retry) — fixed: all four branches (succeed / non-transient-throw / transient-then-success / transient-then-transient) plus the emit-guard are unit-tested at the function boundary.
- N2 (classify the raw error, not the
Session start failed:wrapper) — fixed atsession-start-retry.ts:98(classifyError(String(firstErr))), which restores the non-transient-throw branch. - CLI drift — fixed:
cli/src/types.tsnow mirrorserrorClass?: 'transient' | 'service' | 'user'exactly (same field name, optionality, literals, order) — CLAUDE.md types-sync contract satisfied. - Governance (ADR-003) — cleared: #610 carries the
approvedlabel and covers this PR + the merged #600. (Thepr/branch name is a de-facto-waived nit — five PRs merged onpr/*branches last week.)
I independently re-verified the load-bearing invariants: classifier pattern ordering (specific before catch-all; the (?:agent_status|subtype)= widening matches both wrappers), fail-safe defaults (absent errorClass ⇒ user, isTransientError false ⇒ never auto-retry an unknown), the stuck-guard advisory-only contract (only ever none/steer, never kills), and cancel-wins (triple-guarded: registration order + early-return + dispatcher break). All correct.
One non-blocking finding (below) is worth fixing before the #247 failure-renderer lands, but it changes no live behavior today, so it doesn't gate the merge.
Non-blocking findings
N1 (Medium, but inert today) — the [auto-retried] breadcrumb is never stamped on the one path a retry actually ran-and-failed (branch 4), and the new module's docstring claims the opposite. Independently confirmed by control-flow reading + two agents.
Trace: on branch 4 (transient → retry also fails), startSessionWithRetry throws at session-start-retry.ts:116 and never returns, so orchestrate-task.ts:189 (autoRetried = retried) is unreachable. autoRetried stays false in the outer catch → retriedNote = '' (:222) → the failure persists as Session start failed: <err> with no [auto-retried]. So the marker fires in zero cases where a retry genuinely ran-and-failed — precisely the case it exists to record.
The session-start-retry.ts:64-67 docstring asserts the reverse — "the caller stamps its own [auto-retried] marker … because this function throws rather than returns on the double-failure" — but the function attaches nothing to the thrown error, so the caller cannot observe autoRetried across the throw. The docstring is misleading for the next maintainer.
Why it's non-blocking: the classifier ignores the suffix (no classification impact), and no consumer renders it on main yet — retryGuidance()/[auto-retried] ship ahead of the #247 failure-renderer. So it's a dormant diagnostic-breadcrumb bug, not a live regression. But when #247 lands, a user who hit a double-transient session-start failure would be told "reply to retry" instead of "I already tried again" — the exact confusion the marker was built to prevent. Worth fixing now while the contract is fresh.
- Fix: make the retry fact observable across the throw (attach
autoRetried: trueto the thrown error, or flip the caller's flag via a callback before the second attempt), and add anorchestrate-tasktest asserting the persisted message ends with[auto-retried]on the double-transient path (see inline suggestion). Or, if branch-4-no-stamp is genuinely intended, delete the false docstring paragraph.
N2 (Low) — untested wiring seams (both sides). pr-test-analyzer confirms orchestrate-task.ts:179-224 (does the handler actually call startSessionWithRetry and stamp the marker?) has no test — durableHandler isn't exported and the suite tests only the exported helpers. Commit 6b7cd3f0 covers the agent-side max_turns-enrichment seam, not this CDK retry seam. Agent-side twin: the between-turns hook that writes _LAST_STUCK_SUMMARY (hooks.py:110-111) — the production input to the enrichment — is bypassed by the enrichment test's monkeypatch, so deleting those two lines would leave the latch permanently None and every test stays green. Both are low-severity (the extracted units are well-covered; blast radius is ~5 lines of glue), but they're the seams a future refactor would silently break. Fixing N1 naturally closes the CDK half.
N3 (Low) — retryGuidance() fall-through branches unpinned. The retryable-user (non-guardrail) and not-retryable-user copy (error-classifier.ts:1498-1503) aren't asserted; the high-signal transient/service/guardrail copy is. Two more assertions keep the #247 contract from rotting.
Documentation
No doc/guide/ADR changes required — this is internal classifier/handler/agent logic with no new env var, command, or contract surface. Roadmap not touched (acceptable; error-classifier hardening isn't a roadmap line item). No Starlight mirror impact.
Tests & CI
- CI: all green (CodeQL ×3, dead-code advisory, secrets/deps/actions,
build (agentcore), title);MERGEABLE. - Dead-code ratchet: advisory-only (not in
mise run build,continue-on-errorin CI); theErrorClassTypealias it flags is genuinely used. No action. - Coverage: strong and behavior-first — the four-branch retry matrix with
.rejects.toBe()identity checks (distinguishing "re-throws the original" from "throws the retry's"), the every-classification-carries-an-errorClass invariant loop, thesubtype=+ Exec-format cases, the stuck-guard boundary trio (exact>=/full-window edges, K10 false-positive avoidance), and the max_turns-enrichment truth table. Gaps are the two seams in N2 + N3 copy — all non-blocking. - Bootstrap (ADR-002): N/A — no CDK construct/stack/resource-type change (classifier + handler logic + agent Python only).
- #366 perf: clean — the changed CDK test files are pure unit tests (no
new App()/Template.fromStack()/bundling).
Review agents run (Stage 3 — mandatory)
| Agent | Ran? | Outcome |
|---|---|---|
code-reviewer |
✅ | 1 non-blocking → N1 (branch-4 [auto-retried] never stamped + misleading docstring). Pattern ordering, fail-safe defaults, CLI mirror exactness, advisory-only/cancel-wins all verified correct. |
silent-failure-hunter |
✅ | No blocking silent failures. Retry-emit swallow justified+logged; non-transient re-thrown (original error); double-transient throws the retry's error; stuck-guard fail-open swallows don't alter screening/terminal resolution; max_turns enrichment is inside the crash-handled try (degrades gracefully). Independently reached the same N1. |
pr-test-analyzer |
✅ | Prior B2 closed (4 branches + emit guard); the orchestrate-task retry seam + hooks.py latch-refresh seam still untested (N2); retryGuidance fall-through copy unpinned (N3). #366 clean. |
type-design-analyzer |
⏭️ omitted | ErrorClass/ErrorClassType are simple const-object enums; no complex new type. |
comment-analyzer |
⏭️ omitted | Comment accuracy covered by hand + code-reviewer (N1 is the comment/docstring finding). |
/security-review |
⏭️ omitted | No IAM/Cedar/network/secrets/input-gateway change. |
Human heuristics
- Proportionality — ✅ Matches documented live incidents (ABCA-483/659/662); minimal surface; the stuck-guard is advisory-only, no kill authority.
- Coherence — ✅
ErrorClassunifies retry semantics repo-wide; the CLI mirror stays in lockstep; #600 stacks cleanly. - Clarity —
⚠️ One drift: thesession-start-retry.ts:64-67docstring describes a caller-stamps-the-marker flow the code doesn't implement (N1). - Appropriateness — ✅ Sound; the retry is confined to the one idempotent-by-construction step (no clone/commits/PR yet), and mid-run crashes are deliberately not retried.
Bottom line: approving. Both prior blockers resolved, governance cleared (#610 approved), invariants verified. Please address N1 (make branch 4 stamp [auto-retried] + fix the docstring, ideally with the seam test in N2) before the #247 renderer lands — it's the one finding with a future user-facing edge, though it's inert today.
🤖 Generated with Claude Code
| error: emitErr instanceof Error ? emitErr.message : String(emitErr), | ||
| }); | ||
| } | ||
| const handle = await strategy.startSession(input); |
There was a problem hiding this comment.
Non-blocking (N1) — branch 4 throws here, so the caller never learns a retry happened. This second startSession() throws on the double-transient path, so orchestrate-task.ts:189 (autoRetried = retried) is never reached and the outer catch stamps no [auto-retried] marker — the one case a retry genuinely ran-and-failed produces no breadcrumb. The docstring at :64-67 claims "the caller stamps its own marker … because this function throws", but nothing is attached to the thrown error, so the caller can't observe it. Inert today (no renderer consumes the marker on main), but it silently breaks the contract the #247 failure-renderer will depend on.
Make the fact observable across the throw:
| const handle = await strategy.startSession(input); | |
| try { | |
| const handle = await strategy.startSession(input); | |
| return { handle, autoRetried: true }; | |
| } catch (retryErr) { | |
| // Let the caller stamp `[auto-retried]` even though we throw: the retry | |
| // DID run and fail — that's exactly the case the marker records. | |
| if (retryErr instanceof Error) { | |
| (retryErr as Error & { autoRetried?: boolean }).autoRetried = true; | |
| } | |
| throw retryErr; | |
| } |
Then in orchestrate-task.ts catch: const retried = autoRetried || (err as { autoRetried?: boolean })?.autoRetried === true; and add a handler test asserting the persisted error_message ends with [auto-retried] on the transient→transient path (currently untested — N2).
…st seams N1 (Medium): the [auto-retried] marker was never stamped on branch 4 (transient- then-transient), because startSessionWithRetry THROWS there and the caller's autoRetried flag is only set on the success paths. A double-transient failure was told 'reply to retry' instead of 'I already retried' — the exact confusion the marker exists to prevent. Fix: tag the thrown error with a Symbol (AUTO_RETRIED) + export isAutoRetried(); orchestrate-task reads it in the catch and stamps the marker on both paths a retry ran. Corrected the docstring (was claiming the fact was 'observable via the thrown-from state' — it wasn't). N2 (Low): pin the two untested wiring seams. CDK: branch-4 test now asserts isAutoRetried(err) is true (and false for a first-attempt/non-object failure). Agent: new tests drive _stuck_guard_between_turns_hook end-to-end and assert the _LAST_STUCK_SUMMARY latch is written (and cleared on a healthy window) — the production path the enrichment test's monkeypatch bypasses. Deleting hooks.py's latch write now fails a test. N3 (Low): pin retryGuidance's two USER fall-through branches (retryable-user non-guardrail, not-retryable-user) so the #247 failure-renderer copy contract can't rot. Full cdk (2345) + agent (1251) gates green.
|
Addressed all three nits in `2837130` (you approved, but N1 was worth fixing while the contract is fresh — thanks for the catch): N1 (the medium one) — `[auto-retried]` now stamped on the double-transient path. You were exactly right: branch 4 throws from `startSessionWithRetry`, so the caller's `autoRetried` local was still `false` in the catch and the marker was never applied — a double-transient failure got the first-attempt copy ("reply to retry") instead of "I already retried". Fix: tag the thrown error with a `Symbol(autoRetried)` and export `isAutoRetried(err)`; `orchestrate-task.ts` reads it in the catch (`autoRetried || isAutoRetried(err)`). Corrected the docstring that claimed the fact was already "observable via the thrown-from state". N2 — both wiring seams pinned. CDK: the branch-4 test now asserts `isAutoRetried(secondErr) === true` (+ a negative test: a first-attempt/non-object failure is NOT tagged). Agent: two new tests drive `_stuck_guard_between_turns_hook` end-to-end and assert the `_LAST_STUCK_SUMMARY` latch is written from `guard.recent_failure_summary()` (and cleared on a healthy window) — the production path the enrichment test's monkeypatch bypasses. Deleting `hooks.py:1464-1465` now fails a test. N3 — `retryGuidance` fall-throughs pinned. Added assertions for the retryable-USER (non-guardrail) → "reply with extra guidance" and not-retryable-USER → "a retry may not resolve this" branches, built as explicit classifications so a future reclassification of a sample string can't silently un-cover them. Full cdk (2345) + agent (1251) gates green. (One flaky `Synthesis finished with errors` in a build run did not reproduce across three clean synths — my changes are Lambda-code-only, no synth-structural change.) |
What
Backports the standalone error-classifier hardening from
linear-verceltomain: user-facing failures that previously rendered as a bare "Unexpected error" now get a precise title + actionable remedy, and a new transient/service/user axis drives a safe auto-retry.Three commits:
fix(agent-classifier): recognizeAgent session error (subtype=…)wrapper — the classifier keyed only onagent_status=…, so thesubtype=…wrapper emitted by the terminal-error path fell through to "Unexpected error" (live-caught on ABCA-483: a real max-turns cap surfaced as "Unexpected error").feat(errors): 3-way transient/service/user classification + auto-retry-once on transient session-start — a newErrorClassaxis (transient/service/user) on every classification, because theretryableboolean conflated "self-heals on retry" with "you must change something first".retryGuidance()now keys onerrorClass.orchestrate-task.tsauto-retries once on a transient session-start failure — the one place a retry is idempotent by construction (no clone/commits/PR yet) — and stamps[auto-retried]so the surface can say "I tried again, still failed". Mid-run crashes are not retried.fix: classify the claude Exec-format/broken-shim failure with retry guidance (ABCA-659) — when the agent image can't exec theclaudeCLI (Exec format error/native binary not installed), classify it as a transient COMPUTE/image fault with a retry-then-rebuild remedy instead of "Unexpected error".Scope / backport note
This is one slice of a larger error-feedback arc on
linear-vercel. The rest of that arc modifiesfailure-reply.ts/orchestration-reconciler.ts/orchestration-rollup.ts, which are part of the not-yet-merged #247 orchestration subsystem and don't exist onmain— so they're deliberately left out.Commit 2 is a partial cherry-pick of its origin commit: the origin also touched
failure-reply.ts(#247-only, absent frommain), so that hunk is dropped. The kept portion (error-classifier.ts+orchestrate-task.ts) only depends onclassifyError/isTransientErrorand stands alone. Theorchestrate-task.tsmerge preservesmain'sfailTask(…, task.repo)signature and omits an unrelated#299readOnlyfield thatmain'sstartSessioninput doesn't define.Test
mise run buildgreen incdk/: compile clean, 124 suites / 2273 tests pass, global coverage gate met, eslint--fixproduces no mutations, synth clean (only pre-existing cdk-nag INFO/WARN annotations).