[trajectory-grader] Implement tool-output-consumption-rate

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/tool-output-consumption-rate.md

Implements rank 12 (Tier 2) from the trajectory grader catalog: tool-output-consumption-rate.

Distinct from existing built-in graders: tool-success-rate only measures whether a tool call itself succeeded/failed. This grader instead measures, for observations that succeeded, whether their output was ever actually referenced by a later action (consumedByActionIds non-empty). It surfaces wasted tool calls whose results were fetched but never used — a signal tool-success-rate and the other built-ins do not capture.

IR fields required (per shared/graders/trajectory-ir.md):

• toolCalls[].outputRef — links a tool call to the observation it produced
• observations[].id and observations[].consumedByActionIds — whether/what later action referenced the observation

The grader filters observations[] to those originating from a toolCalls[].outputRef, then computes consumed / total where consumed counts observations with non-empty consumedByActionIds. Returns applicable: false-equivalent (value: null, explanatory message) when no observations or no tool-originated observations are present in the IR.

Catalog: see shared/graders/README.md — 12 of 25 graders now implemented.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 25.7 AIC · ⌖ 9.07 AIC · ⊞ 10.2K · ◷

• expires on Sep 13, 2026, 8:34 AM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33322594191' -n agent -D '/tmp/agent-33322594191'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33322594191/aw-daily-trajectory-grader-tool-output-consumption-rate.bundle'
temp_ref='refs/bundles/create-pr-daily-trajectory-grader-tool-output-consumption-rate-ba17d329f40372dd-d5cee055'
target_ref='refs/heads/daily-trajectory-grader-tool-output-consumption-rate-ba17d329f40372dd'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'daily-trajectory-grader-tool-output-consumption-rate-ba17d329f40372dd'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'daily-trajectory-grader-tool-output-consumption-rate-ba17d329f40372dd'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement tool-output-consumption-rate' --base 'main' --head 'daily-trajectory-grader-tool-output-consumption-rate-ba17d329f40372dd' --repo 'github/gh-aw'