[trajectory-grader] Implement tool-output-consumption-rate

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/tool-output-consumption-rate.md

Implements tool-output-consumption-rate — Tier 2 rank 12, the next unimplemented grader in .github/workflows/shared/graders/README.md (walking tier-first, then rank-within-tier: Tier 1 ranks 5-11 and Tier 2 ranks 1-4 were already done, making rank 12 the next one).

Why distinct from existing built-in graders: tool-output-consumption-rate measures whether tool call results were actually used downstream, not just whether calls succeeded. The built-in tool-success-rate grader only judges call success/failure; it says nothing about whether a successful result was ever referenced by a later action. This grader closes that gap by computing the fraction of IR observations[] entries with a non-empty consumedByActionIds, surfacing wasted exploration (tool calls whose output was fetched but never used).

Required IR fields: observations[] (specifically observations[].consumedByActionIds, as populated per the Trajectory IR build procedure step 4 in trajectory-ir.md).

Output contract: returns { value, unit: "ratio", details } with value: null / not-applicable message when the trace lacks an observations[] array.

See shared/graders/README.md for the full catalog — 8 of 25 graders now implemented.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 26.4 AIC · ⌖ 13.8 AIC · ⊞ 10.2K · ◷

• expires on Sep 13, 2026, 7:34 AM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33319733619' -n agent -D '/tmp/agent-33319733619'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33319733619/aw-trajectory-grader-tool-output-consumption-rate.bundle'
temp_ref='refs/bundles/create-pr-trajectory-grader-tool-output-consumption-rate-6620ed4e4ae70151-81025406'
target_ref='refs/heads/trajectory-grader-tool-output-consumption-rate-6620ed4e4ae70151'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'trajectory-grader-tool-output-consumption-rate-6620ed4e4ae70151'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'trajectory-grader-tool-output-consumption-rate-6620ed4e4ae70151'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement tool-output-consumption-rate' --base 'main' --head 'trajectory-grader-tool-output-consumption-rate-6620ed4e4ae70151' --repo 'github/gh-aw'