tests: fix 5 unrelated CI flakes in pytest/pgaftest harnesses

[dimitri]

Problem

Five separate CI flakes, each diagnosed in an earlier round of triage and left unfixed at the time:

1. pgaftest / multi-misc (PG17) — ensure.pgaf's test_004_demoted timing out.
2. pytest / multi (all PG versions) — test_012_prepare_replication_quorums failing.
3. pytest / citus (PG14–18) — test_001_init_coordinator timing out.
4. pgaftest / ssl (PG15) — test_008_disable_maintenance failing with "container is not running".
5. pgaftest / multi-alternate (PG17) — test_005_002_fail_primary_again failing with "node3 did not pass through assigned-state report_lsn".

None of these are related to each other or to the region/replication-stall work in #1149 — each is an independent timing/synchronization gap in its own corner of the test infrastructure (the old Python harness, the monitor's health-check timing, and the pgaftest tool itself).

Fixes

1. tests/tap/specs/ensure.pgaf — test_003_init_secondary force-restarts node2's Postgres directly (bypassing pg_autoctl) to exercise the keeper's "ensure" restart loop, then confirms node2 self-reports secondary again. That only reflects node2's own FSM converging — the monitor's independent health-check worker (5s probe period, 20s unhealthy-timeout) may not have re-confirmed node2 healthy yet. test_004_demoted immediately kills node1 next; landing inside that window means the monitor can catch node2 mid-unhealthy and refuse to fail over to it. Added a settle sleep 15s between the two steps.

2. tests/test_multi_ifdown.py — test_011_prepare_candidate_priorities changes node2's candidate priority, which makes the monitor rewrite synchronous_standby_names on the primary (node3) via apply_settings — but never waited for node3 to converge back. test_012 then races a primary that's still mid-transition. Added the missing node3.wait_until_state(target_state="primary").

3. tests/pgautofailover_utils.py — wait_until_pg_is_running(timeout=...) accepted a timeout parameter but never forwarded it: neither as --timeout to the underlying pg_autoctl inspect pgsetup wait CLI call (which silently used its own 30s default) nor as the Python-side subprocess-wait timeout (which used execute()'s own 60s default). Citus coordinators can take longer than that under CI load. Now forwards the timeout to both.

4. src/bin/pgaftest/test_runner.c, CMD_COMPOSE_START — docker compose up -d returns as soon as the daemon issues the start, not once the container is registered as actually running; a following exec could race that with "container ... is not running" (confirmed a 75ms gap in raw CI timestamps). Now polls docker inspect --format '{{.State.Running}}' (up to 10s) before declaring success.

5. src/bin/pgaftest/test_runner.c, runner_wait_notify_goal — pass-through state tracking (wait until X state is Y passing through Z) relies on catching a NOTIFY on the monitor's state channel. Two gaps could make it miss a fast intermediate state: the existing periodic 5s SQL-poll fallback only checked the final target state, never backfilling pass-through tracking from what it read; and there was no way to recover a missed state after the fact. Added backfill from the periodic poll's own reads, plus a retroactive check against the monitor's persistent pgautofailover.event log once the wait succeeds (every transition is durably recorded there regardless of whether pgaftest saw it live).

That fix narrowed the flake but didn't eliminate it for one specific case — tests/tap/specs/multi_alternate.pgaf's test_005_002_fail_primary_again. Monitor logs pinned down why: node3's goalState and reportedState can converge on report_lsn and advance to prepare_promotion within the same log timestamp (a ~17µs window observed locally). No observer — live NOTIFY, polling, or a retroactive event-log query — can reliably catch a transition that fast, and it isn't a real FSM bug. Relaxed that one assertion to wait for the stable end state directly, matching how test_003_001/test_003_002 already handle the same kind of transition earlier in the same spec.

Verification

• Style: make docker-check (citus_indent) clean.
• Banned API: ci/banned.h.sh clean.
• make build-pg18 (no-cache): all 12 REGRESS + 6 ISOLATION monitor tests pass — confirms nothing here touches the SQL suite.
• make -C tests run-test TEST=test_multi_ifdown PGVERSION=17: 16/16 passed (fix 2).
• make -C tests run-test TEST=test_basic_citus_operation PGVERSION=17: 19/19 passed (fix 3).
• pgaftest run tests/tap/specs/ensure.pgaf: 5/5 passed (fix 1).
• pgaftest run tests/tap/specs/enable_ssl.pgaf: 8/8 passed (fix 4).
• pgaftest run tests/tap/specs/multi_alternate.pgaf: 16/16 passed, twice in a row (fix 5 + spec relaxation).