Freeze conformance requirements per spec revision, run each at its wire

[felixweinberger]

Adds requirements/<revision>.yaml, a frozen list of the scenarios a spec revision
requires, and runs each revision at that revision's wire version with the server
invocation the SDK's own config declares for that revision. Ships two sets,
2025-11-25 and 2026-07-28; tier-check runs both and Tier 1 means every listed
revision passes.

Fixes #446.

Motivation and Context

Two problems, one mechanism.

The suite accumulates scenarios continuously, so "which scenarios must my SDK
pass to conform to the spec released on 2026-07-28" has no answer today, and an SDK
can drop below 100% while standing still. json-schema-2020-12-preservation merged
three days after the 2026-07-28 spec shipped and four days after 0.2.0-alpha.10
was published; measured against main, typescript-sdk fails it on a commit that
passes 100% against the release it pins. SEP-1730 relegates Tier 1 on any
continuously failing test, so post-release additions start demotion clocks against
SDKs that changed nothing. --spec-version cannot express this: it filters by which
revision a scenario targets, not when it became a requirement.

A revision is also a wire. The dated revisions through 2025-11-25 use the
stateful initialize handshake; 2026-07-28 is stateless with per-request _meta,
and scenarios emit different checks under each. A scenario claimed by both revisions
must therefore run once per revision, on that revision's wire — and the server an
SDK needs may differ per revision. go-sdk only speaks >= 2026-07-28 when its
transport is constructed stateless (its CI starts the server twice); csharp-sdk
serves the stateless lifecycle at /stateless. A single --conformance-server-url
cannot describe either, which is why tier-check builds on src/sdk-runner's
existing per-revision specOverrides instead of inventing a parallel mechanism.

What this adds

# assess a known SDK against everything it claims — clone, build, per-revision
# server, both legs, one verdict
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk

# what does conforming to 2026-07-28 require?
npx @modelcontextprotocol/conformance list --requirements 2026-07-28

# run one revision's set by hand
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --requirements 2026-07-28

• requirements/<revision>.yaml — the frozen contract, covering the two roles the
  spec defines (MCP server as OAuth resource server, MCP client as OAuth client).
  not_scored entries run and are reported but never counted, each with a reason:
  extension (optional per SEP-1730), added-after-release (no implementation
  could have been passing it), or pending (the suite's own reference fixture
  cannot pass it yet — the implementation under test may, so it runs for
  visibility; typescript-sdk main already passes all three dated pending
  scenarios). Every scenario in the suite lands somewhere except the two
  2025-03-26-only legacy backcompat tests, which SEP-1730 excludes unless legacy
  support is claimed. An authorization section is rejected: the spec puts
  authorization-server implementation beyond its own scope.
• --requirements <revision> on client, server and list; the revision pins
  the wire version and replaces --suite/--spec-version/--scenario (conflicts
  are errors, not precedence). sdk gains the same flag, resolved through
  specOverrides.
• tier-check --sdk <name[@ref]> / --sdk-path <dir>: for each claimed revision,
  resolve that revision's build/server/client from known-sdks.ts, manage the
  server, run both legs with cwd = the SDK checkout (so CI-style relative client
  commands work), and merge. --repo/--branch derive from the config.
  --requirements defaults to every shipped revision. The single-URL mode remains
  for SDKs not in the config, with its constraint documented.
• Exit code: nonzero when any scored conformance scenario fails or a leg could not
  be measured; governance findings never affect it. A child run that cannot happen
  (stale set naming a renamed scenario, unreachable server) is reported as
  not measured, never as 0% against the SDK.
• --expected-failures under --requirements judges scored scenarios only, so a
  not-scored failure cannot flag as an unexpected regression.
• Reports label every entry with its revision (tools-list [2026-07-28]); terminal,
  markdown and JSON agree on counts. JSON passed/failed/total describe the scored
  set and always agree with pass_rate; unscored runs are under not_scored.
• 2026-07-28 is a faithful snapshot (anchored to 0.2.0-alpha.10, published the
  day before the revision shipped). 2025-11-25 is a reconstruction from the same
  anchor and its header says so: the release current at its ship date predates
  spec-version awareness entirely.
• Fixes exposed by driving every tier-1 SDK through the new path:
  • csharp's server launches from its build-output directory: ASP.NET reads
    appsettings.json from the content root, and AllowedHosts — which
    dns-rebinding-protection tests — silently never loaded from the repo root.
  • rust's STATELESS=1 override is removed: its CI runs one default server for
    both revisions, and the forced mode failed its SEP-2575 scenarios.
  • withManagedServer waits for the child to actually exit before returning, so
    consecutive runs on one port cannot race a dying predecessor.
  • The tier-check legs spawn their child asynchronously. The previous
    synchronous exec blocked the event loop for a whole leg, so the parent
    stopped draining the managed server's log pipes; a chatty server (rust's
    per-request tracing) filled the 64KB pipe, its writes blocked, and the
    frozen server was scored as 26 conformance failures. The child's stdout is
    dropped entirely (results come from -o), removing the maxBuffer kill for
    verbose SDKs along the way.

How Has This Been Tested?

Driven end to end against local main checkouts of all five tier-1 SDKs:

SDK	Mode	Server	Client
typescript-sdk	URL	67/67	50/50
python-sdk	URL and --sdk-path	67/67	50/50
go-sdk	--sdk-path	67/67	50/50
csharp-sdk	--sdk-path	67/67	50/50
rust-sdk	--sdk-path	67/67	50/50

Each --sdk run demonstrably switches wire per revision (go: stateful then
stateless server process; csharp: / then /stateless). A follow-up adversarial
review (six fresh lenses, refute-by-default verification) drove out and fixed:
skipped legs merging as failures (and the inverse silently passing an unmeasured
revision), the legacy exit-code regression, requirement-file validation (unknown
keys, duplicates, scored∩not_scored), branch attribution in --sdk mode, baseline
auto-attachment under --requirements on the sdk command, duplicate-revision
double counting, and list --requirements omitting pending entries. Negative controls: a
deliberately broken server scores 1/37 with server_conformance blocking and exit
1; a set naming a scenario the build lacks reports not measured naming the
scenario; --requirements with --suite, --scenario, --spec-version, unknown
revisions, comma lists on single-revision commands, and empty values each exit 1
with a message naming the problem. Unit tests pin both requirement sets against the
build (a rename cannot silently shrink a frozen set) and validate the not_scored
taxonomy.

Breaking Changes

None to scenario execution; all flags are opt-in, and tier-check without
--requirements keeps its report-only exit 0. Under --requirements the exit code
gates on scored conformance failures (or an unmeasured leg), which is the CI
contract the frozen sets make meaningful.

Two behaviour changes to disclose for existing consumers:

• tier-check's JSON now reports passed/failed/total over the scored set (so
  they always agree with pass_rate; previously they counted every scenario and
  the object was self-contradictory), and details[].scenario carries canonical
  scenario names rather than timestamped result-directory names.
• Three KNOWN_SDKS entries changed, so conformance sdk results are not
  comparable across this PR for them: rust-sdk's STATELESS=1 override for
  2026-07-28 is removed (its own CI runs one default server for both revisions,
  and the forced mode fails its SEP-2575 scenarios), csharp-sdk's server launches
  from its build-output directory (ASP.NET loads appsettings.json from the
  content root; from the repo root the AllowedHosts filter that
  dns-rebinding-protection tests silently never loaded), and python-sdk's build
  mirrors its CI's two targeted syncs (--all-packages is unbuildable on main).

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have added or updated documentation as needed

Additional context

Deliberately out of scope: a CI freeze check pinning shipped requirement files
(currently a header rule plus review), and promoting 2026-07-28 out of
DRAFT_PROTOCOL_VERSION (shifts every SDK's active suite; separate decision). A
requirement set is this repo's contract — the opposite of an SDK's own
expected-failures baseline, which lives in the SDK repo and records what it knows it
fails; a baselined failure is still a failure against a requirement set, so the two
compose rather than substitute.

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@447

commit: b73c385

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@447

commit: f0120f1