Skip to content

Remove short workflow deadlines from openai_agents tests - #1841

Closed
DABH wants to merge 1 commit into
mainfrom
flake/openai-redundant-deadlines
Closed

Remove short workflow deadlines from openai_agents tests#1841
DABH wants to merge 1 commit into
mainfrom
flake/openai-redundant-deadlines

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What was changed

  • The 5-10s execution_timeout values in tests/contrib/openai_agents/test_openai.py are removed (14 call sites), following [SDK Sentinel] Remove flaky Gemini replay workflow deadline #1816.
  • assert_status_retry_behavior asserts the workflow failure cause chain (ActivityError -> ApplicationError of type APIStatusError) before inspecting history.

Why

On loaded macOS CI runners the first workflow task is occasionally delayed past the default 10s workflow task timeout; the failing jobs log the agent starting 13s after the workflow was started, immediately followed by Evicting workflow ... Error reporting WFT to server. The server retries the task, but by then the workflow's own 10s execution timeout has fired, so test_run_config_models fails with TimeoutError: Workflow timed out (run 34253408987 attempt 1, 3.14/macos-arm) and test_exception_handling gets a WorkflowFailureError with no activity in history and fails on assert found (run 33848127955 attempt 1, 3.10/macos-arm).

These deadlines are nested inside pytest-timeout's 60s bound and serve no other purpose in these tests, so a stalled runner can only turn a healthy workflow into a timed-out one. Dropping them matches #1816; the remaining 30-120s deadlines in the file were not part of the failures and are left alone. Asserting the cause in assert_status_retry_behavior makes an unexpected failure mode show up directly rather than as a missing history event.

Testing

  • The tests use TestModel (no API key); nothing here touches the live-model paths.
  • Emulated an 11s stall of the first workflow task (worker in debug mode with an interceptor blocking the first activation): both tests fail exactly as in CI before the change (TimeoutError('Workflow timed out'), assert found) and pass after it (the retried task runs, the model activity fails with APIStatusError on attempt 2, cause chain as asserted).
  • pytest --flake-finder --flake-runs=30 of both tests alongside a concurrent -n 8 pytest session: all passes (30 runs each; the run covered the eight tests from the same investigation, 240/240 in total, load average 45-53). The unmodified tests also pass 240/240 on the same machine, so the CI failures only reproduce through the stall emulation above.
  • poe lint clean.

Several openai_agents tests started workflows with a 5-10s execution timeout nested inside pytest-timeout's 60s bound. On a loaded macOS CI runner the first workflow task was occasionally delayed past the default 10s workflow task timeout; the retried task then ran after the workflow's own execution timeout had already fired, so test_run_config_models failed with 'TimeoutError: Workflow timed out' and test_exception_handling found no activity in history ('assert found'). Drop those deadlines, as #1816 did for test_replay_tool_loop; pytest-timeout still bounds real hangs. Also assert the failure cause in assert_status_retry_behavior so an unexpected failure mode is reported directly.
@DABH

DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #1835, which now removes the redundant deadlines from both the google_genai and openai_agents tests under one rationale.

@DABH DABH closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant