Skip to content

Freeze conformance requirements per spec revision, run each at its wire #447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
felixweinberger merged 1 commit into from
Aug 7, 2026
Merged

Freeze conformance requirements per spec revision, run each at its wire #447

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 31 additions & 94 deletions .claude/skills/mcp-sdk-tier-audit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@ Two components work together:

## Quick Start: CLI

For an SDK listed in `src/sdk-runner/known-sdks.ts`, tier-check manages the whole conformance side itself — clone/build, then for each shipped revision the server invocation that SDK declares for that revision:

```bash
gh auth login
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk
```

The per-SDK sections below are for the manual path: driving a server you started yourself.

The CLI is a subcommand of the [MCP Conformance](https://github.com/modelcontextprotocol/conformance) tool.

```bash
Expand Down Expand Up @@ -95,56 +104,33 @@ The skill lives in `.claude/skills/` in this repo, so if you open [Claude Code](
3. Run the skill:

```
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd]
/mcp-sdk-tier-audit <local-sdk-path> <conformance-server-url> [client-cmd] [--requirements <revision>]
```

Pass the client command as the third argument to include client conformance testing. If omitted, client conformance is skipped and noted as a gap in the report.

**TypeScript SDK example:**
**Pass `--requirements` with every revision the SDK claims**, comma-separated. Each revision's scenarios run at that revision's own wire version, and all of them must pass for Tier 1: the dated revisions through `2025-11-25` use the stateful initialize handshake while `2026-07-28` is stateless, so a scenario belonging to both has to work on both and one run does not cover the other. It also means a scenario added to the suite after a revision shipped cannot fail an SDK that had no opportunity to adopt it. Without the flag, scoring uses the suite as it stands today, which is not a tier claim. See [Conformance Requirements](../../../README.md#conformance-requirements), and run `conformance list --requirements 2025-11-25,2026-07-28` to see both sets.

```bash
# Terminal 1: start the everything server (build first: npm run build)
cd ~/src/mcp/typescript-sdk && npm run test:conformance:server:run
**Known SDKs — no server to start, no commands to look up:**

# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/typescript-sdk http://localhost:3000/mcp "npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts"
```

**Python SDK example:**

```bash
# Terminal 1: install and start the everything server
cd ~/src/mcp/python-sdk && uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001

# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/python-sdk http://localhost:3001/mcp "uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py"
```

**Go SDK example:**

```bash
# Terminal 1: build and start the everything server
cd ~/src/mcp/go-sdk && go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"

# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/go-sdk http://localhost:3002 "/tmp/go-conformance-client"
/mcp-sdk-tier-audit <sdk checkout> --requirements 2025-11-25,2026-07-28
```

**C# SDK example:**

```bash
# Terminal 1: start the everything server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003
The skill drives `tier-check --sdk-path`, which builds the SDK and starts the
server invocation its config declares **per revision** — the per-SDK commands
live in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts),
not here. That matters because a hand-run example cannot be correct for every
SDK: go-sdk needs a different server process per revision (`-stateless` defaults
to true, so a bare invocation mis-measures 2025-11-25 — [#446](https://github.com/modelcontextprotocol/conformance/issues/446)),
and csharp-sdk serves the two revisions at different endpoints. Prose copies of
those invocations drift; the config is tested.

# Terminal 2: run the audit (from the conformance repo)
/mcp-sdk-tier-audit ~/src/mcp/csharp-sdk http://localhost:3003 "dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient"
```
**An SDK not in `known-sdks.ts`:** start its everything server yourself and pass
the URL and client command explicitly — see [Other SDKs](#running-conformance-tests)
below. The one endpoint you give must serve every claimed revision at its own
wire, or the numbers will be wrong for the revisions it does not speak.

The skill derives `owner/repo` from git remote, runs the CLI, launches parallel evaluations for docs and policy, and writes detailed reports to `results/`.

### Any Other AI Coding Agent

Expand Down Expand Up @@ -181,66 +167,17 @@ Run the CLI for the scorecard, then review docs and policies yourself using the

## Running Conformance Tests

To include conformance test results, start the SDK's everything server first, then pass the URL to the CLI. To also run client conformance tests, pass `--client-cmd` with the command to launch the SDK's conformance client.

**TypeScript SDK**:
For any SDK in [`src/sdk-runner/known-sdks.ts`](../../../src/sdk-runner/known-sdks.ts), don't start servers by hand — the config runs the right invocation per revision:

```bash
# Terminal 1: start the server (SDK must be built first)
cd ~/src/mcp/typescript-sdk && npm run build
npm run test:conformance:server:run # starts on port 3000

# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/typescript-sdk \
--conformance-server-url http://localhost:3000/mcp \
--client-cmd 'npx tsx ~/src/mcp/typescript-sdk/test/conformance/src/everythingClient.ts'
```
# both legs, every revision, one verdict
npx @modelcontextprotocol/conformance tier-check --sdk go-sdk

**Python SDK**:

```bash
# Terminal 1: install and start the server
cd ~/src/mcp/python-sdk
uv sync --frozen --all-extras --package mcp-everything-server
uv run mcp-everything-server --port 3001 # specify port to avoid conflicts

# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/python-sdk \
--conformance-server-url http://localhost:3001/mcp \
--client-cmd 'uv run python ~/src/mcp/python-sdk/.github/actions/conformance/client.py'
# a single leg at a single revision, for debugging
npx @modelcontextprotocol/conformance sdk --path <sdk checkout> --mode server --requirements 2026-07-28
```

**Go SDK**:

```bash
# Terminal 1: build and start the server
cd ~/src/mcp/go-sdk
go build -o /tmp/go-conformance-server ./conformance/everything-server
go build -o /tmp/go-conformance-client ./conformance/everything-client
/tmp/go-conformance-server -http="localhost:3002"

# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/go-sdk \
--conformance-server-url http://localhost:3002 \
--client-cmd '/tmp/go-conformance-client'
```

**C# SDK**:

```bash
# Terminal 1: start the server (requires .NET SDK)
cd ~/src/mcp/csharp-sdk
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework net9.0 -- --urls http://localhost:3003

# Terminal 2: run tier-check (server + client conformance)
npm run --silent tier-check -- \
--repo modelcontextprotocol/csharp-sdk \
--conformance-server-url http://localhost:3003 \
--client-cmd 'dotnet run --project ~/src/mcp/csharp-sdk/tests/ModelContextProtocol.ConformanceClient'
```
To reproduce one leg fully by hand, copy the build/server/client commands from that SDK's `known-sdks.ts` entry (including its `specOverrides` for the revision you're running) rather than from a README: the entries are exercised by the test suite and per-revision, prose examples are neither.

**Other SDKs:** Your SDK needs an "everything server" — an HTTP server implementing the [Streamable HTTP transport](https://modelcontextprotocol.io/specification/draft/basic/transports.md) with all MCP features (tools, resources, prompts, etc.). See the implementations above as reference.

Expand Down
33 changes: 28 additions & 5 deletions .claude/skills/mcp-sdk-tier-audit/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: >-
Produces tier classification (1/2/3) with evidence table, gap list, and
remediation guide. Works for any official MCP SDK (TypeScript, Python, Go,
C#, Java, Kotlin, PHP, Swift, Rust, Ruby).
argument-hint: '<local-path> <conformance-server-url> [client-cmd] [--branch <branch>]'
argument-hint: '<local-path> [conformance-server-url] [client-cmd] [--requirements <revisions>] [--branch <branch>]'
---

# MCP SDK Tier Audit
Expand All @@ -26,7 +26,9 @@ If this fails (exit code non-zero or shows "not logged in"), stop immediately an

Do NOT proceed to any other step if this check fails.

After parsing arguments (Step 1), also verify the conformance server is reachable:
**Known SDKs need no server from you.** When the SDK is in `src/sdk-runner/known-sdks.ts` (typescript-sdk, python-sdk, go-sdk, rust-sdk, csharp-sdk, and variants), skip the server preflight entirely: Step 2's `--sdk-path` mode builds the SDK and starts the right server per revision itself. The reachability check below applies only to the URL fallback for unknown SDKs.

After parsing arguments (Step 1), for the URL fallback only, verify the conformance server is reachable:

```bash
curl -sf <conformance-server-url> -o /dev/null -w '%{http_code}' 2>&1 || true
Expand All @@ -41,8 +43,9 @@ If the server is not reachable, stop and tell the user:
Extract from the user's input:

- **local-path**: absolute path to the SDK checkout (e.g. `~/src/mcp/typescript-sdk`)
- **conformance-server-url**: URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`)
- **conformance-server-url** (only for SDKs not in `known-sdks.ts`): URL where the SDK's everything server is already running (e.g. `http://localhost:3000/mcp`). That one endpoint must serve every claimed revision at its own wire version; SDKs that vary the server per revision cannot be driven this way
- **client-cmd** (optional): command to run the SDK's conformance client (e.g. `npx tsx test/conformance/src/everythingClient.ts`). If not provided, client conformance tests are skipped and noted as a gap in the report.
- **requirements** (optional): spec revisions to score against, comma-separated, e.g. `--requirements 2025-11-25,2026-07-28`. Each revision's scenarios run at that revision's wire version, and every listed revision must pass for Tier 1. Scores the SDK against exactly the scenarios that revision required when it shipped, rather than everything the suite carries today. Prefer it whenever the question is "does this SDK conform to revision X". Without it, scoring uses today's suite, which can fail an SDK for a scenario added after it shipped. Run `conformance list --requirements <revision>` to see the set.
- **branch** (optional): Git branch to check on GitHub (e.g. `--branch fweinberger/v1x-governance-docs`). If not provided, derive from the local checkout's current branch: `cd <local-path> && git rev-parse --abbrev-ref HEAD`. This is passed to the tier-check CLI so that policy signal file checks use the correct branch instead of the repo's default branch.

The first two arguments are required. If either is missing, ask the user to provide it.
Expand All @@ -57,15 +60,35 @@ cd <local-path> && git remote get-url origin | sed 's#.*github.com[:/]##; s#\.gi

The `tier-check` CLI handles all deterministic checks — server conformance, client conformance, labels, triage, P0 resolution, releases, policy signals, and spec tracking. You are already in the conformance repo, so run it directly.

For a known SDK, this manages everything — build, per-revision server, both legs:

```bash
npm run --silent tier-check -- \
--sdk-path <local-path> \
--requirements <revisions> \
--output json
```

`--repo` and `--branch` are derived from the SDK config and checkout; pass them only to override. For an SDK not in `known-sdks.ts`, fall back to driving it directly:

```bash
npm run --silent tier-check -- \
--repo <owner/repo> \
--branch <branch> \
--requirements <revisions> \
--conformance-server-url <conformance-server-url> \
--client-cmd '<client-cmd>' \
--output json
```

Omit `--requirements` only if the user did not name a revision. When it is set the
scorecard reports `requirements_revision`, both pass rates count exactly the
scenarios that revision requires, and anything run but not scored carries a
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
lists every revision scored, and each detail carries the `revision` it came from. Quote the revision
alongside any conformance number, and report the not-scored failures separately
rather than folding them into the score or omitting them.

If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped). The `--branch` flag should always be included (derived from the local checkout if not explicitly provided).

The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
Expand Down Expand Up @@ -117,8 +140,8 @@ Combine the deterministic scorecard (from the CLI) with the evaluation results (

### Tier 1 requires ALL of:

- Server conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Client conformance test pass rate == 100% (date-versioned scenarios only; `draft` and `extension` are informational and not scored)
- Server conformance test pass rate == 100% across every requirement set given to `--requirements` (each run at its own wire), otherwise of date-versioned scenarios only
- Client conformance test pass rate == 100%, on the same basis
- Issue triage compliance >= 90% within 2 business days
- All P0 bugs resolved within 7 days
- Stable release >= 1.0.0 with no pre-release suffix
Expand Down
Loading
Loading