feat(tools): declare preconditions, so a missing id becomes a handoff - #381
Merged
Merged
Conversation
Nothing routes an MCP call: the client's model picks a tool from the descriptions
alone. A tool that needs an identifier the caller does not have is a dead end -
the model asks the user for something they also do not know, or gives up. It does
not reach for a fallback, because no description tells it one exists.
The concrete case that prompted this: "List all projects in my workspace".
- 15 of the 17 Test Management tools require project_identifier
- NONE of them accepts its absence
- NO tool in this server lists projects
So the first step of nearly every Test Management journey was unserved, and the
model had nothing to prefer AND no handoff to follow. It only reached
askBrowserstackAI when a user named it explicitly.
This appends one sentence per tool naming where the missing id comes from. A
sibling tool whenever one can produce it - faster and more predictable than an
agent, exactly as askBrowserstackAI's own description argues. askBrowserstackAI
only where no tool here can:
tm 15 tools -> project id via askBrowserstackAI (nothing lists projects)
3 of those also -> test plan id via listTestPlans
a11y fetchAccessibilityIssues -> scan ids from startAccessibilityScan
getAccessibilityAuthConfig -> id from createAccessibilityAuthConfig
tra listTestIds -> build id via getBuildId / listBuildId
fetchRCA -> test ids via listTestIds
getFailureLogs, fetchSelfHealedSelectors, prepareSelfHealingPlan
-> session id, which nothing lists
Kept as shared constants in tool-handoff.ts rather than per-tool prose: the
wording IS the routing signal, and twenty hand-written variants drift into twenty
different signals.
Deliberately NOT included: a listProjects tool. That would also fix this query
and is arguably the better primitive, but it is a separate decision about the
tool surface; this change is the general fix and helps every tool that needs an
id, not just the project case.
Two things the mechanical pass got wrong and were corrected: the bare tool-name
string also appears inside handlers (trackMCP, error paths), so anchoring on it
rewrote live code - insertion is anchored on server.tool( instead; and in
rca-agent.ts the schema argument is a bare identifier rather than X.shape, so the
first attempt appended to the SCHEMA. Its type error also revealed that fetchRCA
takes testId[], not a build id, so it points at listTestIds.
Full suite green (482 tests), typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Nothing routes an MCP call. The client's model picks a tool from the descriptions alone.
So a tool that needs an identifier the caller doesn't have is a dead end: the model either
asks the user for something they also don't know, or gives up. It does not reach for a
fallback, because no description tells it one exists.
The case that prompted this — "List all projects in my workspace":
project_identifierThe first step of nearly every Test Management journey was unserved. The model had nothing
to prefer and no handoff to follow, so it only reached
askBrowserstackAIwhen a usernamed it explicitly — which is not something a user should have to know.
The change
One sentence appended per tool, naming where the missing id comes from. A sibling tool
whenever one can produce it — faster and more predictable than an agent, exactly as
askBrowserstackAI's own description argues.askBrowserstackAIonly where no tool can.askBrowserstackAI(nothing lists projects)getTestPlan,listSubTestPlans,getSubTestPlanlistTestPlansfetchAccessibilityIssuesstartAccessibilityScan, else the agentgetAccessibilityAuthConfigcreateAccessibilityAuthConfig, else the agentlistTestIdsgetBuildId/listBuildIdfetchRCAlistTestIdsgetFailureLogs,fetchSelfHealedSelectors,prepareSelfHealingPlanComposed,
listTestCasesnow reads:Held as shared constants in
src/tools/tool-handoff.ts, not per-tool prose: the wordingis the routing signal, and twenty hand-written variants drift into twenty different
signals. Rewording later is one edit.
Deliberately not included
A
listProjectstool. It would also fix this query and is arguably the betterprimitive — but it's a separate decision about the tool surface, and this change is the
general fix: it helps every tool that needs an id the caller lacks, not just the project
case. Worth doing as well, not instead.
What this does not fix
Tool choice is a model decision, and it varies between runs — we've seen the same query
land on different tools across two passes. This raises the probability that a stuck model
finds the next step; it cannot guarantee it. Any claim of deterministic routing would need
a different architecture, not better descriptions.
Testing
description argument and not on the schema. Verified programmatically: all 25
insertions follow a string literal, none follows a bare identifier.
Two mechanical mistakes were caught and corrected on the way, both worth knowing if anyone
repeats this pass:
trackMCP, error paths), soanchoring on
"toolName",rewrote live code. Insertion is anchored onserver.tool(.rca-agent.tsthe schema argument is a bare identifier (FETCH_RCA_PARAMS) ratherthan
XSchema.shape, so the first attempt appended the sentence to the schema. Theresulting type error also revealed that
fetchRCAtakestestId[], not a build id — soit points at
listTestIds, which was a genuine content fix, not just a syntax one.Base
Branched off
askrelay/central-mcp(PR #379), because the handoff text namesaskBrowserstackAIand that tool does not exist onmain. If #379 lands first this rebasescleanly; if #379 is deferred, the
askBrowserstackAIpointers need to becomelistProjectsor be dropped, and the sibling-tool pointers still stand on their own.