-
Notifications
You must be signed in to change notification settings - Fork 35
React to CLI update and Anthropic response format changes #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ots for extended thinking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR introduces two improvements to the E2E test infrastructure: preventing corrupted snapshots on test failure and updating snapshots to reflect CLI changes for Anthropic extended thinking compatibility.
Changes:
- Implements test failure detection in each language's test framework to skip writing snapshots when tests fail
- Updates snapshot files to reflect new CLI behavior that coalesces tool calls into single assistant messages for Anthropic extended thinking
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/harness/replayingCapiProxy.ts | Adds support for skipWritingCache parameter to prevent snapshot writes on test failure |
| python/e2e/testharness/proxy.py | Adds skip_writing_cache parameter to stop() method |
| python/e2e/testharness/context.py | Passes test_failed flag to proxy teardown to conditionally skip snapshot writes |
| python/e2e/conftest.py | Implements pytest hook to track test failures and skip snapshot writes |
| nodejs/test/e2e/harness/sdkTestContext.ts | Uses Vitest's onTestFailed hook to track failures and adds COPILOT_CLI_PATH env var support |
| nodejs/test/e2e/harness/CapiProxy.ts | Adds skipWritingCache parameter to stop() method |
| go/e2e/testharness/proxy.go | Adds StopWithOptions method with skipWritingCache parameter |
| go/e2e/testharness/context.go | Uses Go's t.Failed() to detect test failures and skip snapshot writes |
| dotnet/test/Harness/E2ETestContext.cs | Checks CI environment variable to skip snapshot writes (xUnit limitation) |
| dotnet/test/Harness/CapiProxy.cs | Adds skipWritingCache parameter to StopAsync method |
| test/snapshots/tools/invokes_built_in_tools.yaml | Updated snapshot reflecting coalesced tool calls format |
| test/snapshots/session/*.yaml | Updated snapshots with minor variations in assistant responses |
| test/snapshots/permissions/*.yaml | Updated snapshots with coalesced tool calls and response variations |
| test/snapshots/mcp-and-agents/*.yaml | Updated snapshots with response variations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
d82a2c3 to
40de300
Compare
2055997 to
456dc7d
Compare
456dc7d to
4388a4e
Compare
As of updating to latest CLI, the E2E tests failed because of response format changes.
1. Update snapshots for Anthropic extended thinking
The CLI now coalesces tool calls into single assistant messages for Anthropic extended thinking compatibility. This updates all affected snapshots to match the new format.
2. Skip writing snapshots on test failure
Prevents corrupted snapshots from being written when tests fail. Each language uses its native test framework hooks: