test(func-tests): add functional + live smoke harness for ci API commands#1362
Merged
mergify[bot] merged 1 commit intoMay 6, 2026
Conversation
Member
Author
|
This pull request is part of a Mergify stack:
|
Contributor
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 🤖 Continuous IntegrationWonderful, this rule succeeded.
🟢 👀 Review RequirementsWonderful, this rule succeeded.
🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 ReviewsWonderful, this rule succeeded.
🟢 📕 PR descriptionWonderful, this rule succeeded.
|
There was a problem hiding this comment.
Pull request overview
Adds a new functional-test harness that exercises the installed mergify binary end-to-end against a local mock HTTP server, plus a separate “live” smoke layer that hits the real Mergify API on a scheduled workflow to catch mock drift.
Changes:
- Add
func-tests/suite usingpytest-httpserverto validateci scopes-sendandci junit-processwire behavior via subprocess execution of the real CLI binary. - Add a nightly/manual
func-tests-liveworkflow runningpytest -m liveagainst the real API when a token is available; add alivepytest marker. - Wire functional tests into PR CI via a new
func-testsjob and addpytest-httpserverto dev dependencies / lockfiles.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Adds pytest-httpserver and its transitive deps (werkzeug, markupsafe) to the locked dev set. |
| pyproject.toml | Adds pytest-httpserver dev dep, registers live marker, and ruff per-file ignores for func-tests/. |
| poe.toml | Adds poe func-test and poe live-test task entry points. |
| func-tests/conftest.py | Introduces subprocess-based CLI runner fixtures and environment scrubbing; adds live_token fixture. |
| func-tests/test_scopes_send.py | Functional tests for Rust-native ci scopes-send against a mock HTTP server. |
| func-tests/test_junit_process.py | Functional tests for ci junit-process/junit-upload via mock server expectations. |
| func-tests/test_live_smoke.py | Live smoke tests (marked live) targeting real API endpoints. |
| func-tests/README.md | Documents the new functional + live test layers and how to run them. |
| func-tests/fixtures/junit_pass.xml | Passing JUnit fixture for functional/live runs. |
| func-tests/fixtures/junit_fail.xml | Failing JUnit fixture for quarantine-path functional testing. |
| .github/workflows/func-tests-live.yaml | Adds scheduled/manual workflow to run live smoke tests using an environment-scoped secret. |
| .github/workflows/ci.yaml | Adds func-tests job to the PR CI gate. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
bf4ce3f to
949f0c2
Compare
Member
Author
Revision history
|
jd
reviewed
May 5, 2026
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
949f0c2 to
e9507cf
Compare
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
e9507cf to
828db85
Compare
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
828db85 to
ed1af72
Compare
Three smoke tests under func-tests/ that drive the real `mergify`
binary against the real Mergify API, gated on
LIVE_TEST_MERGIFY_TOKEN:
- `ci scopes-send` — POST /v1/repos/{owner}/{repo}/pulls/{n}/scopes
- `ci junit-process` — OTLP traces upload + quarantine check
- `config simulate` — POST /v1/repos/{owner}/{repo}/pulls/{n}/simulator
Each fires when the real API's URL, auth, or wire format diverges
from what the CLI expects. Asserts only "endpoint exists, accepts
our payload, returns 2xx" — never response content, since the
test tenant's state is not under test control.
Driven by `.github/workflows/func-tests-live.yaml` on a nightly
cron + manual dispatch against
mergify-clients-testing/mergify-cli-repo PR #1. Runs in a
dedicated `func-tests-live` GitHub Environment so the
LIVE_TEST_MERGIFY_TOKEN secret can be rotated and audited
independently. NOT wired into the PR ci-gate — an upstream blip
cannot block PRs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Change-Id: I1ee94f6c9e1d6ac7d4ad22fe07b98860ecec12e9
sileht
force-pushed
the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
from
ed1af72 to
e5975fa
Compare
jd
approved these changes
May 5, 2026
remyduthu
approved these changes
May 6, 2026
Contributor
Merge Queue Status
This pull request spent 11 minutes 17 seconds in the queue, including 10 minutes 59 seconds running CI. Waiting for:
All conditions
ReasonThe merge conditions cannot be satisfied due to failing checks Failing checks:
HintYou may have to fix your CI before adding the pull request to the queue again. |
36 tasks
Member
Author
|
@Mergifyio queue |
Contributor
Merge Queue Status
This pull request spent 11 minutes 51 seconds in the queue, including 11 minutes 1 second running CI. Required conditions to merge
|
36 tasks
mergify
Bot
deleted the
devs/sileht/func-tests/add-functional-live-smoke-harness-ci-api-cmds--1ee94f6c
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two new test layers parallel to compat-tests:
Mock-based functional tests under func-tests/ that drive the real
mergify binary against pytest-httpserver and assert request /
response shape for
ci scopes-send(4 cases) andci junit-process(5 cases). Wired into ci-gate via a new
func-testsjob inci.yaml.
Live smoke tests (test_live_smoke.py, marked
pytest.mark.live)that hit the real Mergify API at
mergify-clients-testing/mergify-cli-repo PR build(deps): Bump rich from 10.2.0 to 10.2.1 #1. Skipped unless
LIVE_TEST_MERGIFY_TOKEN is set; driven by a dedicated
func-tests-live workflow on a nightly cron + manual dispatch, NOT
wired into ci-gate. Catches the class of mock drift schema-level
checks miss — real auth, real serialization, response shape.
The mock layer keeps the PR feedback loop fast and offline-safe; the
live layer is a canary for "the mock has lied about the wire
contract." The live workflow runs in a dedicated
func-tests-liveGitHub Environment so the staging-tenant secret can be rotated and
audited independently.
Adds pytest-httpserver==1.1.3 to dev deps and a
livepytest marker.Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com