[trajectory-grader] Implement tool-output-consumption-rate

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/tool-output-consumption-rate.md

Implements tool-output-consumption-rate — rank 12, Tier 2 (needs explicit provenance/reference IDs) — the next unimplemented grader in the catalog per tier-first, then rank-within-tier walk order (11 of 25 previously implemented; Tier 1 fully done, this is the first Tier 2 grader beyond ranks 1–4).

What it answers: of everything the agent fetched via a tool call, how much of it was actually used later? It computes the fraction of observations[] entries with a recorded sourceToolCallId whose consumedByActionIds is non-empty.

Why it's distinct from existing built-ins: tool-success-rate measures whether calls succeeded; trajectory-efficiency measures redundant/loop steps. Neither captures a call that succeeds but whose result is never read or acted upon — wasted fetch work that this grader surfaces.

Required IR fields: observations[].sourceToolCallId and observations[].consumedByActionIds, per the canonical Trajectory IR (shared/graders/trajectory-ir.md). Reports not-applicable when the trace lacks observations or none carry a sourceToolCallId, rather than fabricating a value.

Catalog and remaining count: see shared/graders/README.md — 12 of 25 implemented after this PR.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 29.3 AIC · ⌖ 12.7 AIC · ⊞ 10.2K · ◷

• expires on Sep 13, 2026, 8:03 AM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33321123046' -n agent -D '/tmp/agent-33321123046'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33321123046/aw-add-tool-output-consumption-rate-grader.bundle'
temp_ref='refs/bundles/create-pr-add-tool-output-consumption-rate-grader-2017b5d9c7260367-0ef52f76'
target_ref='refs/heads/add-tool-output-consumption-rate-grader-2017b5d9c7260367'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'add-tool-output-consumption-rate-grader-2017b5d9c7260367'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'add-tool-output-consumption-rate-grader-2017b5d9c7260367'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement tool-output-consumption-rate' --base 'main' --head 'add-tool-output-consumption-rate-grader-2017b5d9c7260367' --repo 'github/gh-aw'