Skip to content

Speed up test DBs and reduce memory footprint#10167

Draft
stephanos wants to merge 1 commit intomainfrom
stephanos/tweak-databases
Draft

Speed up test DBs and reduce memory footprint#10167
stephanos wants to merge 1 commit intomainfrom
stephanos/tweak-databases

Conversation

@stephanos
Copy link
Copy Markdown
Contributor

Combination of independent experiments validated on parallel feature/tweak-db-* branches under standard ARM runners. Per-job memory peaks were captured by the upgraded memory_monitor.sh.

Changes:

  • Healthcheck interval 3s -> 1s (cheap; fast services like Postgres no longer wait an extra tick).
  • Cassandra:
    • MAX_HEAP_SIZE 2G -> 1G (HEAP_NEWSIZE 200M -> 100M). Per-job Cassandra container peak: 2586MB -> 1533MB (-41%). No runtime regression on cass_* jobs; cass_es8 -10% on test runtime.
    • JVM flags to skip the ~30s gossip-settle wait that's pointless on a single-node test cluster (ring_delay_ms=1000, skip_wait_for_gossip_to_settle, single-token settings).
  • Elasticsearch / OpenSearch heap 1G -> 512M. Per-job ES7 peak 1404MB -> 852MB (-40%); ES8 1464MB -> 943MB (-36%). System peak on ES-using jobs dropped 13.9GB -> 11.6GB median (-17%). No runtime regression observed.
  • Postgres fsync=off, synchronous_commit=off, full_page_writes=off; MySQL --innodb-flush-log-at-trx-commit=0, --innodb-doublewrite=0, --skip-log-bin. CI databases are ephemeral so durability has no value but fsync overhead is real. Test runtime: mysql8 -23%, postgres12 -26%.
  • memory_monitor.sh now tracks per-container memory via docker stats and prints a system+per-container peak summary at run end (visible at the bottom of every CI job log).

Combined wall-clock saving on shard jobs alone: ~2000s across the full matrix. Stability either equal or improved (5/40 failures vs 10/43 baseline on the standard runners; 8-core runners should clear those timeout-driven retries).

What changed?

Describe what has changed in this PR.

Why?

Tell your future self why have you made these changes.

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Potential risks

Any change is risky. Identify all risks you are aware of. If none, remove this section.

@stephanos stephanos force-pushed the stephanos/tweak-databases branch 8 times, most recently from 857e6ab to cbda309 Compare May 3, 2026 20:53
@stephanos stephanos force-pushed the stephanos/tweak-databases branch from cbda309 to 64b6390 Compare May 3, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant