Skip to content

Reset the exit code when the test harness finishes a passing run - #3040

Merged
nohwnd merged 2 commits into
mainfrom
fix-child-exit-code-leaking-into-ci
Sep 5, 2026
Merged

Reset the exit code when the test harness finishes a passing run#3040
nohwnd merged 2 commits into
mainfrom
fix-child-exit-code-leaking-into-ci

Conversation

@nohwnd

@nohwnd nohwnd commented Sep 5, 2026

Copy link
Copy Markdown
Member

test.ps1 could end with a non-zero $LASTEXITCODE while every test passed. CI ends the step with exit $LASTEXITCODE, so a fully green run goes red with no failing test in it. Which test ran last decides whether it happens, which is what would make it intermittent.

The cause is that Invoke-InNewProcess calls the child as a native command, so the child's exit code lands in the caller's $LASTEXITCODE, and several tests run a child that exits non-zero on purpose.

What I got wrong first, since the PR history shows it

My first attempt made the helper restore the caller's value. That is wrong: propagating the child's exit code is intended and tested. Exitcode is set to the number of failed tests and the process exits when tests fail asserts $LASTEXITCODE | Verify-Equal 1 in the caller after the child ran, and two more tests assert 0. CI failed on every leg and was right to.

So the helper is untouched, and the fix is one place: test.ps1 sets the exit code to 0 once it has decided the run passed.

It cannot hide a failure

test.ps1 does not decide pass/fail from $LASTEXITCODE:

if (0 -lt $result.Count) { ... if ($CI) { exit 1 } }   # P tests failed
if ("Failed" -eq $r.Result) { throw "Run failed!" }     # Pester run failed

Both come before the reset and both override it. The bug is one-directional: green runs going red, never red runs going green.

No test

This is the harness's own exit path, so there is nothing to assert on from inside the suite it runs. The behaviour it protects, the child's code reaching the caller, is already covered by the three tests named above, and those now pass unchanged.

What this does not claim

main went red once today on PS7 - Ubuntu 22.04 with exit code 1, every suite green and no error text, and a re-run passed. This is the right shape but I could not prove it was the cause: the known leaking test leaves 2, not 1, and the tests added in #3020 do not make their child exit non-zero (checked, the child ends on a string so it exits 0 before and after that change). If a silent non-zero exit shows up again after this lands, that rules this class out.

🤖

Invoke-InNewProcess calls the child as a native command, which overwrites
$LASTEXITCODE in the parent, and nothing put it back. Several tests here run a
child that exits non-zero on purpose, and they read that code from the child's
own output, not from ours.

So test.ps1 could end with a non-zero $LASTEXITCODE while every test passed. CI
ends the step with `exit $LASTEXITCODE`, which turns a fully green run red with no
failing test in it. Which test ran last decided whether it happened, which is what
would make it intermittent.

Measured against main: a child that exits 3 left the caller on 3, now it stays 0.

The helper restores the caller's value in a finally, and test.ps1 sets the exit
code to 0 once it has decided the run passed. That second one is the safety net for
any native command added to the suite later. Both sit after the existing `exit 1`
and `throw "Run failed!"` paths, so neither can hide a real failure. The bug was
one-directional: green runs going red, never red runs going green.

Added a test that fails against main.
Invoke-InNewProcess propagating the child's exit code is intended and tested:
"Exitcode is set to the number of failed tests and the process exits when tests
fail" asserts $LASTEXITCODE is 1 in the caller after the child ran, and two more
tests assert it is 0. Restoring it in the helper broke that contract, and CI caught
it on every leg.

So the helper goes back to what it was, and the fix is only the reset at the end of
test.ps1, after both failure paths. Whichever test ran last still decides what
$LASTEXITCODE holds when the harness finishes, and this stops that leaking into
CI's `exit $LASTEXITCODE`.

Also drops the test I added, it asserted the opposite of the contract above.
@nohwnd nohwnd changed the title Stop a child process's exit code leaking into the test harness Reset the exit code when the test harness finishes a passing run Sep 5, 2026
@nohwnd
nohwnd enabled auto-merge (squash) September 5, 2026 14:58
@nohwnd
nohwnd merged commit c2ca84f into main Sep 5, 2026
14 checks passed
@nohwnd
nohwnd deleted the fix-child-exit-code-leaking-into-ci branch September 5, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant