[audit-workflows] 🔍 Agentic Workflow Audit — 2026-06-16 (prod-main 84.4%; day-10 sdk-lockout + NEW upload_assets false-red)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-06-16

Window: ~3.3h evening cluster (18:41–21:57Z). A ~20h stretch (06-15 22:00 → 06-16 18:41) is unobserved because the logs MCP tool hit its 120s timeout — a recurring data-coverage limit, so today's totals are partial.

Headline: Overall completion rate 71.2% (47✅ / 19❌ of 66 completed) — but that number is dragged down by a single PR's container-validation batch. Prod-main (main branch) was 84.4% (27✅ / 4❌), and all 4 prod-main failures are known/recurring classes with no new systemic incident.

Metric	Value
Runs (with summary)	67 (66 completed)
Success / Failure	47 / 19
Overall rate	71.2%
Prod-main rate	84.4% (27/32)
Tokens (observed)	41.9M · 894 turns · 750 action-min
Engines	copilot 42 · claude 14 · codex 5 · gemini/antigravity/pi 1 each
missing_tools / missing_data / mcp_failures	0 / 0 / 0
Firewall blocked	197 / 4314 (4.6%) — all by-design

---

📊 Trend Charts (30 days)

Health holds in the mid-80s%/low-90s% band typical of the last two weeks; today's 71.2% dip is a window artifact (one PR's smoke/container batch), not a fleet regression — the 06-13 trough (68.4%) was the real low and rebounded. Failure counts stay small on prod-main.

Observed daily tokens (41.9M) sit near the 7-day moving average; the 06-12 spike (127M) was a full-window outlier and bars track window length, not a usage surge. No token-budget-429 events this window.

---

🚨 Failure Breakdown (19)

Cluster A — PR container-validation batch (9, by-design, NOT prod)

All 9 are on branch copilot/add-gh-aw-node-to-default-containers — a PR that modifies the default container images. Every smoke/container test on that branch failed at agent startup (0-tok / 0-turn): Agent Container Smoke Test, Changeset Generator, Smoke {Claude, Codex, Gemini, Antigravity}, Smoke Copilot ×3 (incl. AOAI apikey/Entra). This is expected PR-validation noise while the container change is iterated — the Agent Container Smoke Test literally checks whether node/tools exist in the container. No prod-main impact.

Prod-main #1 — copilot-sdk tool-permission lockout (2 runs, RECUR day 10)

• Daily Safe Output Integrator (27643760095) — 5 denials of routine commands (read on /pkg/*, shell go version, git status) → SDK aborts at the 5-denial cap, ~24min wall, 0 tokens.
• Daily Testify Uber Super Expert (27641213363) — NEW affected workflow, identical signature, ~26min, 0 tokens (previously failed via the long-gone minimatch class).

This is the longest-running unresolved prod-main class (since 06-02, now day 10). The copilot-sdk-driver path denies read-only/source-inspection commands the workflows legitimately need, then hard-aborts with no retry. Top-priority fix remains: grant the required tool perms or fix the SDK allowlist plumbing.

Prod-main #2 — ⭐ NEW: upload_assets job reddens a fully-successful run (2 runs)

• Daily Code Metrics and Trend Tracking Agent (27643027159, claude) — agent SUCCESS (53 turns, 4.27M tok); activation, agent, detection, safe_outputs, update_cache_memory, push_repo_memory all success — yet the run is red because upload_assets: failure (charts quality_score_breakdown.png + historical_trends.png).
• Daily Project Performance Summary Generator (MCP Scripts) (27649783375, copilot) — same shape: agent SUCCESS, only upload_assets failed.

A new variant of the safe-output partial-failure-intolerance family, isolated to the chart-upload job. Notably the same upload_asset path every chart-emitting report agent uses fleet-wide — including this audit agent (I hit a related path-resolution quirk below). Recommend making upload_assets non-fatal to overall run conclusion (skip-and-warn) or adding upload retry/validation.

Other PR-branch failures (6, non-prod)

• PR Code Quality Reviewer ×3 — only the activation job failed (the daily_effective_workflow_exceeded evening-cap gate) → agent skipped → PR shows red. Known daily-effective-workflow-cap-reddens-pr-runs (UX false-red).
• PR Code Quality Reviewer ×1 (copilot, 1 turn) — sdk tool-perm pattern on support-array-object-runner-specs.
• Test Quality Sentinel (copilot, 1.5M tok / 41 turns) — copilot-harness redden-with-valid-work.
• Design Decision Gate (claude, 1.07M tok / 19 turns) — isolated PR-branch failure.

---

💡 Recommendations

1. [HIGH · day 10] Resolve copilot-sdk-driver tool-permission lockout (now hitting Daily Safe Output Integrator and Daily Testify). Longest-running prod-main class — grant required read/shell perms or fix the SDK allowlist. (rec-sdk-toolperm-allowlist-or-relax-guard)
2. [MEDIUM · NEW] Make the upload_assets job non-fatal — a transient chart-upload failure should not discard a fully-successful agent run. Broad blast radius (every chart-emitting report agent). (rec-upload-assets-nonfatal)
3. [LOW · infra] Two tooling-doc mismatches surfaced this run: the logs tool's 120s timeout keeps leaving ~20h unobserved daily; and upload_asset only resolves workspace-relative paths, not the /tmp/gh-aw/python/charts path the Python skill documents (charts had to be copied into the workspace to upload).

✅ Health Signals

• 0 missing-tool reports, 0 missing-data, 0 MCP failures this window.
• Firewall: 197/4314 (4.6%) blocked — entirely by-design (smoke-test egress probes hitting google/safebrowsing telemetry + Slide Deck Maintainer github-cloud asset pull).
• Repo memory updated: metrics-summary, known-issues (sdk-lockout→day10, +upload_assets-redden), recommendations, anomalies, workflow-trends, audit-history.

References: §27643760095 · §27643027159 · §27641213363

Generated by 🔍 Agentic Workflow Audit Agent · ◷

• expires on Jun 17, 2026, 2:17 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #39907.