test(func-tests): drop mock layer, keep live smoke + add config simulate#1363
Closed
Conversation
…ands Two new test layers parallel to compat-tests: - Mock-based functional tests under func-tests/ that drive the real mergify binary against pytest-httpserver and assert request / response shape for `ci scopes-send` (4 cases) and `ci junit-process` (5 cases). Wired into ci-gate via a new `func-tests` job in ci.yaml. - Live smoke tests (test_live_smoke.py, marked `pytest.mark.live`) that hit the real Mergify API at mergify-clients-testing/mergify-cli-repo PR #1. Skipped unless LIVE_TEST_MERGIFY_TOKEN is set; driven by a dedicated func-tests-live workflow on a nightly cron + manual dispatch, NOT wired into ci-gate. Catches the class of mock drift schema-level checks miss — real auth, real serialization, response shape. The mock layer keeps the PR feedback loop fast and offline-safe; the live layer is a canary for "the mock has lied about the wire contract." The live workflow runs in a dedicated `func-tests-live` GitHub Environment so the staging-tenant secret can be rotated and audited independently. Adds pytest-httpserver==1.1.3 to dev deps and a `live` pytest marker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Change-Id: I1ee94f6c9e1d6ac7d4ad22fe07b98860ecec12e9
The mock layer added in the previous commit duplicated coverage
already provided by the in-tree Rust wiremock unit tests; keeping
it added a pytest-httpserver dependency and a parallel PR-gate job
for no incremental signal.
Removed:
- test_scopes_send.py / test_junit_process.py (mock-based)
- fixtures/junit_fail.xml (only used by deleted tests)
- pytest-httpserver dev dep
- `func-test` poe task and the `func-tests` PR-gate job
Added:
- test_config_simulate — third live smoke test covering
POST /v1/repos/{owner}/{repo}/pulls/{n}/simulator
func-tests/ is now live-only: three smoke tests gated on
LIVE_TEST_MERGIFY_TOKEN, run from the dedicated nightly workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Change-Id: Ia8144945e0c4f1227b075c8614aa5dfa5dc1c3c1
Member
Author
|
This pull request is part of a Mergify stack:
|
Contributor
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 ⛓️ Depends-On RequirementsWaiting for:
This rule is failing.Requirement based on the presence of
🔴 👀 Review RequirementsWaiting for:
This rule is failing.
🟠 🤖 Continuous IntegrationWaiting for:
Waiting checks:
|
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
bf4ce3f to
949f0c2
Compare
sileht
deleted the
devs/sileht/func-tests/drop-mock-layer-keep-live-smoke-add-config--a8144945
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The mock layer added in the previous commit duplicated coverage
already provided by the in-tree Rust wiremock unit tests; keeping
it added a pytest-httpserver dependency and a parallel PR-gate job
for no incremental signal.
Removed:
func-testpoe task and thefunc-testsPR-gate jobAdded:
POST /v1/repos/{owner}/{repo}/pulls/{n}/simulator
func-tests/ is now live-only: three smoke tests gated on
LIVE_TEST_MERGIFY_TOKEN, run from the dedicated nightly workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
Depends-On: #1362