Skip to content

fix(test): create ModelTests before concurrent pool workers - #6040

Merged
cmgoffena13 merged 3 commits into
SQLMesh:mainfrom
devtechedge:fix/concurrent-test-create-before-pool
Sep 10, 2026
Merged

fix(test): create ModelTests before concurrent pool workers#6040
cmgoffena13 merged 3 commits into
SQLMesh:mainfrom
devtechedge:fix/concurrent-test-create-before-pool

Conversation

@devtechedge

@devtechedge devtechedge commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

When sqlmesh test runs with concurrent_tasks > 1 and a unit test sets vars.execution_time, ModelTest.create_test() was running on worker threads. That path can call to_datetime() / ttl_cache (time.time()) while another worker is starting or stopping a time_machine freeze, which produces IndexError: list index out of range (or similar) during test creation.

This moves ModelTest.create_test() onto the calling thread before submitting work to the pool. Workers only run the already-built tests. Creation stays inside the same try/finally that closes engine adapters so an invalid create_test still cleans up connections.

Fixes #6039

Test plan

  • pytest tests/core/test_test.py::test_freeze_time_concurrent -v passed 20 consecutive times locally (race; single pass is not enough)
  • CI green

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

ModelTest.create_test() can call to_datetime()/ttl_cache (time.time())
while another worker freezes time via time_machine for execution_time.
Building tests on the main thread removes that race.

Fixes SQLMesh#6039

Signed-off-by: devtechedge <devtechedge@users.noreply.github.com>

@cmgoffena13 cmgoffena13 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty close, I didn't notice connections being created. So right after connections are created we should make sure all relevant code is in a try statement to properly cleanup connections. Thanks for picking this up so quickly!

P.S. Don't forget our PR checklist. I added it to your description.

Comment thread sqlmesh/core/test/runner.py
Comment thread sqlmesh/core/test/runner.py Outdated
Move calling-thread create_test into the same try/finally that closes
engine adapters so invalid tests still clean up connections. Add a NOTE
about a possible future parallel create stage that must not overlap runs.

Signed-off-by: devtechedge <devtechedge@users.noreply.github.com>
Signed-off-by: Dev M <devtechedge@gmail.com>
@devtechedge

Copy link
Copy Markdown
Contributor Author

Thanks @cmgoffena13 - addressed your review feedback in 4d515a7:

  1. Moved the calling-thread create_test loop into the same try/finally that closes engine adapters, so invalid creates still clean up connections.
  2. Added a NOTE that create could be a separate parallel stage later, as long as it does not overlap with runs.

Also updated the PR checklist for the style/tests/DCO items I ran.

@cmgoffena13
cmgoffena13 self-requested a review September 10, 2026 00:17
@cmgoffena13
cmgoffena13 merged commit 11ac31a into SQLMesh:main Sep 10, 2026
34 checks passed
@devtechedge

Copy link
Copy Markdown
Contributor Author

Thanks @cmgoffena13 for the review and for taking the time to walk through the connection cleanup and concurrency details.

The feedback around keeping create_test within the same try/finally scope was especially helpful. It made me think more carefully about the resource lifecycle when test creation itself can fail. I also appreciated the note about keeping future parallel test creation separate from test execution.

This was a really helpful learning experience for me, both from the concurrency side and from seeing how the implementation should account for cleanup and future extensibility.

Glad we could get this merged and close out #6039. Thanks again for the guidance and review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent model tests crash when execution_time is set

2 participants