Cloud Topics: Deflake L0 GC integration tests and also consolidate them significantly

[oleiman]

Test suite got a little hairy over time and was flaky on CDT.

Remove 7 tests redundant with C++ unit tests or low integration value:

• test_l0_gc: subsumed by every other test that produces + waits for GC
• test_concurrent_pause_start: state machine concurrency covered by
  level_zero_gc_mt_test.cc concurrent_reset_start_pause
• test_reset_while_paused: covered by level_zero_gc_tests.cc
  ResetWhilePaused (exact same scenario)
• test_reset_while_running: covered by level_zero_gc_tests.cc
  ResetWhileRunning (exact same scenario)
• test_grace_period_enforcement: grace period filtering covered by
  level_zero_gc_tests.cc NoDeletesForYoungObjects
• test_epoch_lag_and_catchup: tests internal epoch_lag metric response
  to hot-reconfigure of gc_interval; no cluster invariant at stake
• test_leadership_transfer_during_gc: L0 GC scans the object store by
  prefix range, not by partition ownership; leadership is irrelevant

Combine 5 admin API tests (get_status, basic_pause_unpause,
single_node_pause_unpause, not_found, partial_failure) into one
test_admin_api that exercises all endpoints after a single produce.

Combine test_node_failure_mid_gc and test_topic_deletion_during_gc into
test_cluster_disruption_resilience.

Fold test_gc_metrics batching assertion into test_gc_stress.

Fixes

JIRA filter

• fixes CORE-15919
• fixes CORE-15863
• fixes CORE-15845
• fixes CORE-15809
• fixes CORE-15783
• fixes CORE-15773
• fixes CORE-15772
• fixes CORE-15765
• fixes CORE-15674
• fixes CORE-15590

Proof (CDT)

• AWS x 6
• Azure x 20
• GCP x 10

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v26.1.x
• v25.3.x
• v25.2.x

Release Notes

• none

[oleiman]

/ci-repeat 1
skip-units
dt-repeat=10
tests/rptest/tests/cloud_topics/l0_gc_test.py

[vbotbuildovich]

Retry command for Build#82404

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cloud_topics/l0_gc_test.py::CloudTopicsL0GCResilienceTest.test_cluster_disruption_resilience@{"cloud_storage_type":1}
tests/rptest/tests/cloud_topics/l0_gc_test.py::CloudTopicsL0GCResilienceTest.test_cluster_disruption_resilience@{"cloud_storage_type":2}

[vbotbuildovich]

CI test results

test results on build#82404

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
CloudTopicsL0GCResilienceTest	test_cluster_disruption_resilience	{"cloud_storage_type": 2}	integration	https://buildkite.com/redpanda/redpanda/builds/82404#019d2db8-d20f-4b61-9387-42021f2fcdec	FLAKY	18/20	The test was found to be new, and no failures are allowed	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=CloudTopicsL0GCResilienceTest&test_method=test_cluster_disruption_resilience
CloudTopicsL0GCResilienceTest	test_cluster_disruption_resilience	{"cloud_storage_type": 1}	integration	https://buildkite.com/redpanda/redpanda/builds/82404#019d2db5-0d4b-46ab-91d7-95011f813f65	FLAKY	19/20	The test was found to be new, and no failures are allowed	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=CloudTopicsL0GCResilienceTest&test_method=test_cluster_disruption_resilience

test results on build#82436

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
WriteCachingFailureInjectionE2ETest	test_crash_all	{"use_transactions": false}	integration	https://buildkite.com/redpanda/redpanda/builds/82436#019d3128-8a76-4894-914c-9899de9b3bcb	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0908, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.2485, p1=0.0575, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteCachingFailureInjectionE2ETest&test_method=test_crash_all

test results on build#82455

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
ControllerLogLimitMirrorMakerTests	test_mirror_maker_with_limits	null	integration	https://buildkite.com/redpanda/redpanda/builds/82455#019d38c8-0b88-48f5-a993-e26295556d7e	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0273, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ControllerLogLimitMirrorMakerTests&test_method=test_mirror_maker_with_limits

[oleiman]

/cdt
dt-repeat=20
tests/rptest/tests/cloud_topics/l0_gc_test.py

[Copilot]

Pull request overview

This PR refactors the Cloud Topics L0 GC integration tests to reduce flakiness and significantly consolidate coverage (as described: 21 tests → 8), while adjusting test workloads and timeouts to better align with epoch-based GC behavior.

Changes:

• Reworks produce_some to run for a minimum duration (based on the configured epoch increment interval) to ensure L0 objects span multiple epochs for GC eligibility.
• Consolidates multiple admin API / pause-unpause / partial-failure checks into a single combined test flow.
• Adds/adjusts longer timeouts and introduces a resilience test covering node failure + topic deletion, plus updates data-integrity/stress tests to be more tolerant to transient produce failures.

[oleiman]

/cdt
dt-repeat=50
tests/rptest/tests/cloud_topics/l0_gc_test.py::CloudTopicsL0GCDataIntegrityTest

[claude]

PR Review: Cloud Topics L0 GC test consolidation and deflaking

Overall: Good consolidation — 21 tests down to 8 is a meaningful reduction in CI cost and flake surface. The structural changes (time-based produce_some, wait_for_status helper, try/finally for producer/consumer cleanup, tolerate_failed_produce) are all solid deflaking patterns. The code reads well.

What looks good

• produce_some rework: Switching from message-count-based progress to time-based "run across N epoch intervals" directly addresses the root cause of flakes — GC needs data spread across epochs, not just a count of messages. The wait_until_with_progress_check with progress_sec is a good safeguard.
• wait_for_status helper: Cleanly handles the transient SAFETY_BLOCKED states that were likely causing flakes in direct check_statuses calls.
• try/finally cleanup: Both CloudTopicsL0GCDataIntegrityTest and CloudTopicsL0GCStressTest now properly stop() and free() producer/consumer in finally blocks, preventing resource leaks on assertion failures.
• Timeout increases (30s → 60s): Reasonable given these are integration tests running against real cloud storage.
• Batching assertion moved to stress test: Makes sense — it needs enough data volume to be meaningful.

Concerns

1. Coverage loss in test_admin_api Phase 4 (single-node pause): The old test_single_node_pause_unpause verified the behavioral effect of pausing — that the paused node stopped deleting, and that unpausing caused it to start again. The consolidated test only checks API status, not actual deletion behavior. See inline comment.

2. Coverage loss in Phase 5 (partial failure recovery): After restarting the killed node, the test pauses and starts it but doesn't verify the node actually resumes GC work. The old test_partial_failure also had a gc_pause → restart → gc_pause(recovered_node) flow, but this new version doesn't confirm end-to-end recovery.

3. Removed tests with no replacement:

   • test_concurrent_pause_start (rapid pause/start toggling with concurrent resets) — this was a concurrency stress test. Is this coverage still needed, or is it considered low-value?
   • test_reset_while_paused and test_reset_while_running — gc_reset() is no longer exercised anywhere in the file.
   • test_grace_period_enforcement — verified the minimum-object-age grace period. Is this covered elsewhere?
   • test_epoch_lag_and_catchup — verified hot-reconfiguration of GC intervals. Is this covered elsewhere?
   • test_leadership_transfer_during_gc — verified GC survives leadership transfers.
   • test_gc_metrics (dedicated metrics test) — some metrics checks moved to stress test, but epoch_lag, max_deleted_epoch, and objects_listed metrics are no longer tested.

   It would be helpful to note in the PR description which of these removals are intentional coverage reductions (low-value/flaky) vs. coverage that's been folded into the remaining tests.

4. Minor: The test_l0_gc basic test was removed entirely — the simplest "produce, wait for deletes" smoke test is gone. The closest replacement is test_idle_housekeeping, but that has a different purpose (idle GC after repeater stops).

Nits

• See inline comments for minor style/consistency observations.

No blocking issues — the deflaking improvements are valuable and the clean CDT run is encouraging. The main question is whether the removed concurrency/reset/grace-period/leadership-transfer coverage is intentionally dropped or should be tracked for follow-up.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[oleiman]

/cdt
rp_version=build
provider=azure
region=centralus
cdt_instance_type=Standard_L8as_v3
dt-repeat=20
tests/rptest/tests/cloud_topics/l0_gc_test.py

[oleiman]

/cdt
provider=gcp
region=us-west2
cdt_instance_type=n2d-standard-4
dt-repeat=10
tests/rptest/tests/cloud_topics/l0_gc_test.py

[claude]

PR Review: Cloud Topics L0 GC Test Consolidation

Overall this is a solid cleanup that significantly reduces test suite size (-893/+342 lines) while preserving meaningful integration coverage. The consolidation rationale in the PR description is well-justified — each removed test is mapped to its C++ unit test counterpart. The CDT proof across AWS/Azure/GCP with multiple runs is thorough.

What looks good

• produce_some refactor: The time-based approach with wait_until_with_progress_check is much more robust than the old message-count approach. Ensuring production spans multiple epoch intervals directly addresses the root cause of flakiness (GC needs objects spread across epochs).
• wait_for_status helper: Clean polling abstraction that replaces repeated inline check_statuses + wait_until patterns.
• Resource cleanup: Adding try/finally with producer.free() / consumer.free() in both test_no_data_loss_under_gc and test_gc_stress fixes resource leaks on assertion failures — the old code didn't call free() at all.
• Timeout bumps (30s → 60s): Sensible for CI environments; the old 30s timeouts were likely a flake source.
• test_admin_api consolidation: Combining 5 small admin API tests into phased execution after a single produce cycle is a good trade-off — the phases are well-labeled and logically ordered.
• test_cluster_disruption_resilience: Merging node failure + topic deletion into one test with a longer produce phase makes sense since both test GC's resilience to cluster disruptions.

Suggestions

• Weak assertion in stress test (line 1024): assert acked > 0 is very permissive for a test that's supposed to push ~3 GiB. If the producer mostly fails, downstream GC/L1 assertions would fail with confusing errors. Consider a minimum threshold (e.g., MSG_COUNT // 2) like the data integrity test uses.
• Edge case in produce_some: If _epoch_increment_interval_ms < 1000, integer division would yield epoch_interval_s=0 and min_runtime_s=0, making the time-gate trivially satisfied. Not a concern with current defaults (2000ms) but worth noting.

No concerns with

• Removing tests covered by C++ unit tests (concurrent_pause_start, reset_while_paused/running, grace_period, epoch_lag, leadership_transfer) — the justification is clear and the unit test coverage is cited.
• Adding tolerate_failed_produce=True — appropriate for integration tests where transient produce failures shouldn't cause flakes, especially with the 75% threshold guard in the data integrity test.

[WillemKauf]

deflakes LGTM, haven't spend a lot of time ensuring that we aren't losing test coverage in removal of the ducktape tests, but I'm willing to take the commit & cover letter at face value knowing that these things have been author & bot validated

[vbotbuildovich]

/backport v26.1.x

[oleiman]

deflakes LGTM, haven't spend a lot of time ensuring that we aren't losing test coverage in removal of the ducktape tests, but I'm willing to take the commit & cover letter at face value knowing that these things have been author & bot validated

Thanks. Can always reintroduce less bad versions of these tests if needed.

[dotnwat]

👍