[MNT] Dockerized tests for CI runs using localhost

[satvshr]

Metadata

• Reference Issue: fixes [MNT] Intermediate test plan #1614, stacks on [ENH] Allow using a local test server #1630
• New Tests Added: No
• Documentation Updated: No

Details

• What does this PR implement/fix? Explain your changes.
  This PR implements the setting up of the v1 and v2 test servers in CI using docker via localhost.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.72%. Comparing base (39daaef) to head (27d3aba).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1629      +/-   ##
==========================================
- Coverage   53.07%   52.72%   -0.35%     
==========================================
  Files          37       37              
  Lines        4381     4381              
==========================================
- Hits         2325     2310      -15     
- Misses       2056     2071      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[geetu040]

I don't think any failing test is coming from this PR. It would be better to conditionally skip them and link #1657. If there is new failure message which is not already mentioned there, then please comment down the failure with the failing tests so it could be tracked there. Also if some tests are failing because of pandas, create a separate issue for that, skip and link to these then.

[geetu040]

Looks good, this has not been addressed yet #1629 (comment)

[geetu040]

There are so many changes not related to the issue. These must be coming from upgraded pre-commit in your local setting. Can you undo these changes? This PR should touch only the docker related part.

[geetu040]

The PR looks good, tests are also passing. I will approve this as #1629 (review) is addressed.

[geetu040]

Is it complete from your side? Looks fine to me.

Also are we sure we have skipped all the right tests?

I will also take a look at the skips, match them with reported tests locally and on the issue #1657

[satvshr]

Is it complete from your side? Looks fine to me.

Yes, it is.

Also are we sure we have skipped all the right tests?

I did not use your issue as reference but rather the failures which I observed in this PR alone, in the links mentioned here in the comment I left on the issue itself.

I will also take a look at the skips, match them with reported tests locally and on the issue #1657

I have updated the comment linked above on #1657 but feel free to cross-verify if you want to.

[geetu040]

Great work on this.

Tests that pass on the remote test server but fail on the local test server are currently skipped, and we're tracking those in #1657

FYI: running the docker servers in CI takes less than 3 minutes and the tests that previously relied on remote-test-server now using the local-test-server are much faster and reliable since the probability for race-conditions is highly reduced.

@fkiraly @PGijsbers please review and merge.

[PGijsbers]

I looked at this PR just now and I think we should first investigate the reason for the failure of the tests. Has anyone tried this locally? Because when I try it locally it works (spin up services, point python to localhost:8000 and run tests). There are a few small exceptions, but the vast majority of marked tests pass. This makes me think it is rather an issue with the deployment on GH CI, perhaps there are issues with write permissions or so.

[geetu040]

Has anyone tried this locally?

@PGijsbers Yes, they fail locally as well.

I have added some details on the failures in #1657

Quoting from above

Could it be a problem with the arff-to-pq service in docker-compose of openml/services? Since the image mentioned openml-arff-to-pq-converter-cron is not available as a public frozen docker image on docker-hub and therefore the service fallbacks to use the context given in config/arff-to-pq-converter for normal user (non openml-dev team with access to the private image).
But this is just my guess, it can be totally irrelevant to the failure.