Seed based on commit SHA

[adamchainz]

Description

Suggestion from chat with @graingert and @seporaitis: add a --randomly-seed-vcs or similar option that uses the current Git commit SHA as the seed. That way, parallel test runs for a given commit, such as with pytest-split, use the same seed, but it still changes over time.

[seporaitis]

I like the idea of feeding a consistent seed into pytest-randomly, it would definitely help, but I think it should not be linked to a VCS.

One of the appeals of pytest-randomly is the ability to note the seed, and restart the test suite, then come back and fix the flakes later.

Pinning the seed to a commit hash would make the test fail consistently and there the escape hatch would not exist (talking about ci environments here).

I think what would work much better is if pytest-randomly could read the seed from an environment variable. This way it's up to the caller to set it, and e.g. on github actions one could use run id + an attempt number, or store the seed as artifact and reuse it (thinking about my own use case here).

Having written above, I'm also wondering if there's even a clean solution to this. If I use pytest-split and have a matrix of GitHub action jobs, the only way for me to retry a single instance of a job is by pinning the seed, otherwise I'd be running a different set of tests on a reattempt. Perhaps the solution to this is more in pytest-split and not in pytest-randomly

[adamchainz]

One of the appeals of pytest-randomly is the ability to note the seed, and restart the test suite, then come back and fix the flakes later.

I think this option would still allow that, it would still output a --randomly-seed= value to copy-paste.

My thinking now would be a --randomly-seed=git option to extract the current git SHA. I think this would optimize the the pytest-split case, not requiring any further coordination to set the seed between the split runs.

I think what would work much better is if pytest-randomly could read the seed from an environment variable.

I don't see how that's any more convenient than writing --randomly-seed=$MYSEED in your CI workflow or whatever.

Pinning the seed to a commit hash would make the test fail consistently and there the escape hatch would not exist (talking about ci environments here).

This is true. A rebuild of a given SHA would fail just the same.

Perhaps it would be better for the pytest-split case to seed based on current CI run ID, such as ${{ github.run_id }} (docs) on GitHub Actions. In that case, there would be nothing to do in pytest-randomly, as you can use: pytest --randomly-seed=${{ github.run_id }}.