Python: discard pending State on failed/cancelled superstep (#7859) - #8306
Python: discard pending State on failed/cancelled superstep (#7859)#8306LI (ktz03) wants to merge 2 commits into
Conversation
Call State.discard() on runner failure and cancellation paths so staged writes cannot leak into a later successful run on the same Workflow. Fixes microsoft#7859.
agent-framework-automation
Bot
added
the
python
There was a problem hiding this comment.
🟡 Changes recommended
Cleanup can still be skipped during event streaming and cancellation outside the polling block.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Prevents failed workflow supersteps from leaking staged Python state into later runs.
Changes:
- Discards pending state on failure and cancellation paths.
- Adds a workflow-reuse regression test for failures.
File summaries
| File | Description |
|---|---|
_runner.py |
Adds pending-state cleanup paths. |
test_runner.py |
Tests state isolation after failure. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
- Scope cancel/BaseException cleanup across the full superstep until commit - Discard pending state before yielding failure events - Add cancellation regression that stages state then reuses the workflow
| if not iteration_task.done(): | ||
| iteration_task.cancel() | ||
| with contextlib.suppress(asyncio.CancelledError): | ||
| await iteration_task | ||
| if not committed: | ||
| self._state.discard() |
There was a problem hiding this comment.
What happens if cancellation triggers executor cleanup that raises? At _runner.py:181-186, await iteration_task propagates that cleanup error before discard() executes, so the staged writes survive and the next run commits them. I reproduced this with an executor that sets secret and raises RuntimeError from finally. Could discard() run before the await or from a finally block so executor cleanup cannot bypass it?
There was a problem hiding this comment.
Following up on the earlier failure-event comment: could we keep the run reserved until runner cleanup finishes? _runner.py:135-151 can yield a live executor_failed event while iteration_task is still cancelling siblings, so dropping the ResponseStream releases the weak run lock before discard() runs. A successor then reaches the run-kwargs commit and commits the failed superstep state; I reproduced secret: stale at this commit. Could cleanup complete before reuse is allowed, while still permitting a never-started stream to be dropped?
Motivation & Context
State.set()stages writes in_pending, but the runner never calledState.discard()on failure/cancellation. Because aWorkflowreuses the sameRunnerImpl/Stateacrossrun()calls, a later successful run couldcommit()stale pending writes from a prior failed superstep.Fixes #7859.
Description & Review Guide
self._state.discard()on the failed-iteration and cancellation paths inRunnerImpl.run_until_convergence().set_state, then succeeds on a second run and asserts the leaked key is absent from committed state.Related Issue
Fixes #7859
No other open PR targets this issue.
Contribution Checklist