Improve bigmem tests

[serhiy-storchaka]

BPO	31695
Nosy	@pitrou, @vstinner, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-10-04.19:22:19.577>
labels = ['type-feature', 'tests']
title = 'Improve bigmem tests'
updated_at = <Date 2017-10-07.15:45:09.628>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-10-07.15:45:09.628>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-10-04.19:22:19.577>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31695
keywords = []
message_count = 2.0
messages = ['303729', '303885']
nosy_count = 3.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31695'
versions = []

Linked PRs

• gh-75876: Use functools.wraps() in test.support test decorators #155262
• [3.15] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155290
• [3.14] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155291
• gh-75876: Run bigmem tests in a subprocess #155302
• gh-75876: Correct the memory use declared by some bigmem tests #155307
• [3.13] gh-75876: Use functools.wraps() in test.support test decorators (GH-155262) #155311
• [3.15] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155312
• [3.14] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155313
• [3.13] gh-75876: Correct the memory use declared by some bigmem tests (GH-155307) #155314
• [3.14] gh-75876: Run bigmem tests in a subprocess (GH-155302) #156013
• [3.13] gh-75876: Run bigmem tests in a subprocess (GH-155302) #156014
• gh-75876: Limit the address space of a bigmem test #156023
• gh-75876: Report the peak memory use of a bigmem test #156024

[serhiy-storchaka]

I think that the process of running bigmem tests should be improved.

1. Every bigmem test should run in separate process. It's effect on memory fragmentation shouldn't affect other tests.

2. Only one bigmem test should be run at the same time. Otherwise several parallel bigmem tests can exhaust RAM and lead to swapping. In multiprocessing mode the process running a bigmem test should send a signal to other processes, wait until they finished their current tests and paused, run a bigmem test, and send a signal to other processes that they can continue.

3. The hard limit of addressed memory should be set for every bigmem test, according to the declared requirements. It is better to crash one test and report the failure, than hang on swapping.

4. All other tests should be run with the limit set to 2 GiB (or 1 GiB?). This will help to find memory consuming tests which should be made bigmem tests.

5. The maxrss of the process running a bigmem test should be reported in verbose mode after finishing a test.

[pitrou]

+1 to all this :-)

[cmaloney]

Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest. With that also fix 5 above, printing the maxrss after finishing each test.

To support that I'd like to add a private C extension function that uses platform-specific APIs to get process memory usage and update _MemoryWatchdog / memory_watchdog.py to use that information in its print. That makes it so the platform-specific code lives entirely inside C conditional compilation.

The code currently pre-opens the /proc file and passes that through as stdin to reduce PID recycling issues if the parent exits. Unfortunately there is no cross-platform pidfd type mechanism to make this really simple. Instead the new code relies on a pipe to the parent to determine "has the parent been lost" which lets us detect parent process termination and prevent issues if the PID is recycled. It also means we should exit more quickly when the parent intentionally exits and gives a good point to add the end of test print.

[vstinner]

This issue has not been touched for 9 years, I suggest you opening a new issue for your idea. We can add a private function in a test module such as _testcapi. I didn't understand the pidfd issue, I should see the code to understand :-)

[StanFromIreland]

Currently on our buildbot the test takes over one and a half hours, while some of that is unavoidable, it would be good to decrease it a little if possible.

[vstinner]

Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest.

I modified test.libregrtest to log the memory usage with a new get_process_memory_usage() function in test.libregrtest.utils. The function works on Linux, Windows, macOS and FreeBSD.

I also modified @support.bigmemtest to reuse this get_process_memory_usage() function.

Example:

Using random seed: 2542641886
0:00:00 load avg: 2.46 mem: 23.8 MiB Run 505 tests in parallel using 8 worker processes (timeout: 1 hour 40 min, worker timeout: 1 hour 45 min)
(...)
0:22:04 load avg: 5.86 mem: 14.8 GiB [485/505/1] test_descr passed -- running (3): test_bigmem (22 min 3 sec), test_lzma (22 min 3 sec), test_hashlib (4 min 8 sec)
0:22:05 load avg: 5.86 mem: 14.8 GiB [486/505/1] test.test_asyncio.test_streams passed -- running (3): test_bigmem (22 min 4 sec), test_lzma (22 min 4 sec), test_hashlib (4 min 8 sec)
0:22:06 load avg: 5.86 mem: 17.2 GiB [487/505/1] test_mimetypes passed -- running (3): test_bigmem (22 min 5 sec), test_lzma (22 min 5 sec), test_hashlib (4 min 9 sec)
(...)
0:41:07 load avg: 2.82 mem: 30.7 GiB [504/505/1] test_lzma passed (41 min 6 sec) -- running (1): test_bigmem (41 min 6 sec)
0:41:38 load avg: 2.29 mem: 30.1 GiB running (1): test_bigmem (41 min 37 sec)
(...)
1:31:12 load avg: 1.52 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min)
1:31:44 load avg: 1.62 mem: 30.1 GiB running (1): test_bigmem (1 hour 31 min)
1:32:14 load avg: 1.72 mem: 16.1 GiB running (1): test_bigmem (1 hour 32 min)
1:32:45 load avg: 1.43 mem: 4.4 GiB running (1): test_bigmem (1 hour 32 min)
1:32:51 load avg: 1.48 mem: 74.4 MiB [505/505/1] test_bigmem passed (1 hour 32 min)

The mem: 14.8 GiB part is new!