Reset the exit code when the test harness finishes a passing run - #3040
Merged
Conversation
Invoke-InNewProcess calls the child as a native command, which overwrites $LASTEXITCODE in the parent, and nothing put it back. Several tests here run a child that exits non-zero on purpose, and they read that code from the child's own output, not from ours. So test.ps1 could end with a non-zero $LASTEXITCODE while every test passed. CI ends the step with `exit $LASTEXITCODE`, which turns a fully green run red with no failing test in it. Which test ran last decided whether it happened, which is what would make it intermittent. Measured against main: a child that exits 3 left the caller on 3, now it stays 0. The helper restores the caller's value in a finally, and test.ps1 sets the exit code to 0 once it has decided the run passed. That second one is the safety net for any native command added to the suite later. Both sit after the existing `exit 1` and `throw "Run failed!"` paths, so neither can hide a real failure. The bug was one-directional: green runs going red, never red runs going green. Added a test that fails against main.
Invoke-InNewProcess propagating the child's exit code is intended and tested: "Exitcode is set to the number of failed tests and the process exits when tests fail" asserts $LASTEXITCODE is 1 in the caller after the child ran, and two more tests assert it is 0. Restoring it in the helper broke that contract, and CI caught it on every leg. So the helper goes back to what it was, and the fix is only the reset at the end of test.ps1, after both failure paths. Whichever test ran last still decides what $LASTEXITCODE holds when the harness finishes, and this stops that leaking into CI's `exit $LASTEXITCODE`. Also drops the test I added, it asserted the opposite of the contract above.
nohwnd
enabled auto-merge (squash)
September 5, 2026 14:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test.ps1could end with a non-zero$LASTEXITCODEwhile every test passed. CI ends the step withexit $LASTEXITCODE, so a fully green run goes red with no failing test in it. Which test ran last decides whether it happens, which is what would make it intermittent.The cause is that
Invoke-InNewProcesscalls the child as a native command, so the child's exit code lands in the caller's$LASTEXITCODE, and several tests run a child that exits non-zero on purpose.What I got wrong first, since the PR history shows it
My first attempt made the helper restore the caller's value. That is wrong: propagating the child's exit code is intended and tested.
Exitcode is set to the number of failed tests and the process exits when tests failasserts$LASTEXITCODE | Verify-Equal 1in the caller after the child ran, and two more tests assert0. CI failed on every leg and was right to.So the helper is untouched, and the fix is one place:
test.ps1sets the exit code to0once it has decided the run passed.It cannot hide a failure
test.ps1does not decide pass/fail from$LASTEXITCODE:Both come before the reset and both override it. The bug is one-directional: green runs going red, never red runs going green.
No test
This is the harness's own exit path, so there is nothing to assert on from inside the suite it runs. The behaviour it protects, the child's code reaching the caller, is already covered by the three tests named above, and those now pass unchanged.
What this does not claim
mainwent red once today onPS7 - Ubuntu 22.04with exit code 1, every suite green and no error text, and a re-run passed. This is the right shape but I could not prove it was the cause: the known leaking test leaves2, not1, and the tests added in #3020 do not make their child exit non-zero (checked, the child ends on a string so it exits 0 before and after that change). If a silent non-zero exit shows up again after this lands, that rules this class out.🤖