Different test order

[milssky]

Python Version

3.12.11

pytest Version

8.4.2

Package Version

4.0.1

Description

Hey!

I have a strange thing going on (or I don't quite understand how this plugin works).

In github CI I have this order of tests

============================= test session starts ==============================
platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.6.0
Using --randomly-seed=439041728
django: version: 5.2.6, settings: server.settings (from ini)
rootdir: /code
configfile: setup.cfg
plugins: anyio-4.11.0, randomly-4.0.1, Faker-10.0.0, django-test-migrations-1.5.0, timeout-2.4.0, django-4.11.1, cov-7.0.0, django-fakery-4.1.3, httpx-0.35.0, mock-3.15.1, hypothesis-6.140.2
timeout: 10.0s
timeout method: signal
timeout func_only: False
collected 93 items

tests/test_server/test_urls.py .......                                   [  7%]
tests/test_apps/test_tilda/test_repos/test_project.py ..                 [  9%]
tests/test_server/test_admin.py ....................................     [ 48%]
tests/test_apps/test_tgbot/test_entrypoints/test_curator.py ...........  [ 60%]
server/apps/tgbot/states/helper.py .                                     [ 61%]
tests/test_apps/test_tgbot/test_storage.py ..........                    [ 72%]
tests/test_server/test_common/test_transaction_atomic.py .               [ 73%]
tests/test_apps/test_tgbot/test_models.py .                              [ 74%]
tests/test_apps/test_tilda/test_repos/test_project_participant.py .....  [ 79%]
tests/test_apps/test_llm/test_http.py .                                  [ 80%]
tests/test_apps/test_tilda/test_migrations/test_project_data.py ..       [ 82%]
tests/test_apps/test_tgbot/test_views.py .                               [ 83%]
tests/test_apps/test_notificate/test_service.py FF..F.                   [ 90%]
tests/test_apps/test_tilda/test_models.py ...                            [ 93%]
tests/test_apps/test_tgbot/test_migrations/test_statemessage_data.py ..  [ 95%]
tests/test_server/test_logging.py .                                      [ 96%]
tests/test_apps/test_tgbot/test_repos.py ...                             [100%]

I run the same tests locally, in the same docker container with the same key, and get a different test order.

================================ test session starts =================================
platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.6.0
django: version: 5.2.6, settings: server.settings (from ini)
Using --randomly-seed=439041728
rootdir: /code
configfile: setup.cfg
plugins: anyio-4.11.0, Faker-10.0.0, django-fakery-4.1.3, django-test-migrations-1.5.0, hypothesis-6.140.2, cov-7.0.0, django-4.11.1, mock-3.15.1, timeout-2.4.0, randomly-4.0.1, httpx-0.35.0
timeout: 10.0s
timeout method: signal
timeout func_only: False
collected 93 items

tests/test_server/test_urls.py ...                                             [  3%]
tests/test_apps/test_tilda/test_repos/test_project.py ..                       [  5%]
tests/test_server/test_admin.py ....................................           [ 44%]
tests/test_apps/test_tgbot/test_entrypoints/test_curator.py ...........        [ 55%]
tests/test_apps/test_tgbot/test_storage.py ..........                          [ 66%]
tests/test_apps/test_tgbot/test_models.py .                                    [ 67%]
tests/test_apps/test_tilda/test_repos/test_project_participant.py .....        [ 73%]
tests/test_apps/test_tgbot/test_views.py .                                     [ 74%]
tests/test_apps/test_notificate/test_service.py ......                         [ 80%]
tests/test_apps/test_tilda/test_models.py ...                                  [ 83%]
tests/test_apps/test_tgbot/test_repos.py ...                                   [ 87%]
tests/test_apps/test_tilda/test_migrations/test_project_data.py ..             [ 89%]
tests/test_apps/test_tgbot/test_migrations/test_statemessage_data.py ..        [ 91%]
tests/test_server/test_urls.py ....                                            [ 95%]
server/apps/tgbot/states/helper.py .                                           [ 96%]
tests/test_server/test_common/test_transaction_atomic.py .                     [ 97%]
tests/test_apps/test_llm/test_http.py .                                        [ 98%]
tests/test_server/test_logging.py .                                            [100%]

Is this the norm or a bug?

[adamchainz]

Seems like a bug, but it may be caused by someting outside of pytest-randomly's control…

I think you'll need to debug to see what could be different between the two instances. I'm guessing you don't run your tests on CI using a docker container built to the same Dockerfile as local, right?

[milssky]

Oh no. I'm actually running everything in a container built from the same Dockerfile.

What exactly can be debugged?

[adamchainz]

Sorry for no reply.

Logging wise, I would start with a forked version of pytest-randomly with print statements to check what happens in pytest_collection_modifyitems.

On second reading, I noticed that pytest-django is loaded in a different order relative to pytest-randomly. That plugin has a pytest_collection_modifyitems hook as well which may be the culprit for the differing sort orders. The question then becomes why the plugins are loaded in a different order, and if there's anything we could do (make our hook a wrapper?).