Separate-process archive cache lease test flakes when a child misses its heartbeat window #2520

Problem

tests/playground-custom-archive-cache-process.test.ts is timing-fragile and fails
intermittently on saturated CI runners, unrelated to the change under test. It surfaced on
an unrelated feature PR (#2519) as:

[smoke] FAIL (11/16) tests/playground-custom-archive-cache-process.test.ts
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
0 !== 1
    at tests/playground-custom-archive-cache-process.test.ts:28:10

Observed failure: integration job 106431679800

Cause

The test spawns its lease children with WP_CODEBOX_PLAYGROUND_CUSTOM_ARCHIVE_LEASE_MS=300
(:80, :103). heartbeatLease() renews on Math.max(50, Math.floor(leaseMs / 3))
(packages/runtime-playground/src/playground-wordpress-archive-cache.ts:427), so the
surviving child has only a three-tick, 300 ms margin.

If that child is descheduled for longer than its lease — ordinary CPU or IO starvation on a
runner executing the integration group at --concurrency=2 alongside PHP-WASM workloads —
inspectLeaseFile() sees expiresAt <= now (:591). Because the assertions call
maintainPlaygroundCustomArchiveCache(..., { mode: "apply" }), the sweep does not merely
report the lease inactive, it unlinks the reference (:592). referenceCount drops to
0 and the loss is unrecoverable within the test, so line 28 fails 0 !== 1.

The test has two requirements that are in direct conflict on a single lease duration:

• Lines 23–35 require the surviving child's lease to not expire.
• Lines 36–39 require a crashed child's lease to expire quickly (waitForReferenceExpiry).

The current test resolves that conflict by choosing a lease short enough for the second
requirement, which leaves the first with no margin against scheduler jitter.

Confirmation that this is environmental, not a regression

Deterministic reproduction, using SIGSTOP on the surviving child's process group to emulate
a deschedule longer than its lease, then running the assertion from line 28 verbatim:

branch   stall=0      line-28 PASS (refcount 1)
main     stall=0      line-28 PASS (refcount 1)
branch   stall=400    line-28 FAIL actual=0 expected=1
main     stall=400    line-28 FAIL actual=0 expected=1
branch   stall=800    line-28 FAIL actual=0 expected=1
main     stall=800    line-28 FAIL actual=0 expected=1

The failure reproduces identically on unmodified origin/main. 60 unstalled interleaved runs
(20 unloaded, 20 under CPU contention, 20 pinned to two cores at load average 25) passed on
both origin/main and the feature branch, confirming the failure is scheduling-dependent
rather than branch-dependent.

Desired outcome

Make the separate-process lease assertions independent of whether a child process is
scheduled within its lease window, without weakening what the test protects: shared
cross-process archive protection, bounded crashed-process leases, and reclaim after expiry.

Note that a sweep in apply mode deletes a lease it judges expired, so a single missed
heartbeat is not a transient read — it permanently destroys the reference. Any fix should
either decouple the liveness window from the expiry window (independent lease durations per
child, so the surviving child's lease cannot expire from jitter while the crashed child's
still expires promptly), or synchronize the parent with the child's heartbeat instead of
racing it.

Acceptance criteria

• The liveness assertions (:23, :28, :34) cannot fail because of ordinary scheduler or
  IO jitter on a loaded runner.
• The crashed-process lease still expires on a bounded timeline and the archive still becomes
  reclaimable (:36–:39).
• The existing assertions are preserved, not relaxed, skipped, or retried into passing.
• The cross-process sharing, bounded-lease, and lock-ownership coverage remains behavioral.

AI assistance

Extra Chill Bot via OpenCode isolated the failure on PR #2519, proved it was not a regression
from that PR, and characterized the heartbeat/lease margin and the apply-mode unlink that
makes it unrecoverable, under Chris Huber's direction.