fix(ios): budget cold toolchain probes for the first-exec signature stall - #2423
fix(ios): budget cold toolchain probes for the first-exec signature stall#2423thymikee wants to merge 3 commits into
Conversation
Size Report
Startup median (7 runs, lower is better):
|
|
Fixed the eager-closure-budgets failure (666283b). Root cause: Fix: deleted Verified locally against merge-base
|
|
The runner probes need to honor the remaining request budget at 666283b. Three synchronous probes can each run twice for 30 seconds, blocking for roughly 180 seconds without a deadline or cancellation check. Please pass the owning budget through, cap each attempt by its remaining time, and test exhausted and canceled requests; the current immediate-throw mocks do not prove the deadline. Both retry classifiers also inspect error text. The subprocess timeout already carries structured timeout details. Please classify that signal instead, and test that a non-timeout error with similar wording is not retried. |
…tall xcodebuild/xcrun toolchain probes in cache-identity.ts and runner-cache-metadata.ts were budgeted for a warm toolchain (10s/5s), below the ~18-19s syspolicyd signature-verification stall on the first exec after a fresh macOS host boots. Share one 30s floor constant between both call sites and retry once after a timeout while the deadline allows, since the second exec is instant. Closes #2422
…ared module toolchain-probe-budget.ts sat outside every platform-apple facade's eager closure, but runner-cache-metadata.ts (imported from it) sits inside all seven -- so the new import added one module to each, tripping the eager-closure-budgets no-growth gate (#2422). Delete the shared module. cache-identity.ts keeps the canonical constant inline (it was already outside the gated closures); runner-cache-metadata.ts declares its own copy, guarded by a new unit test that asserts the two stay equal.
Three synchronous probes could each retry once at 30 s, so a wedged toolchain host blocked a request for ~180 s with no deadline and no cancellation check. The runner cache decision now takes the owning request's budget (remaining ms + abort signal) and builds one clock per fingerprint read: every attempt runs at min(per-call ceiling, remaining), the retry is skipped once the budget is spent, an exhausted budget fails the decision without starting another probe, and an aborted signal surfaces the cancellation instead of retrying. `ensureXctestrunArtifact` passes the build budget and signal, session reuse passes the startup budget and the request signal, and lease adoption passes the startup budget; a caller with neither is still capped at 45 s total, so the worst case falls from ~180 s to 45 s. Error codes, texts, and the probe hint are unchanged. Both retry classifiers now read the exec layer's structured timeout detail instead of matching "timed out after Nms" in the message. The predicate is exported once from host-kit's command surface and reaches `runner-cache-metadata.ts` through the Apple runner host port, so the file's eager closure is unchanged. Tests use a fake clock that only advances when a probe actually blocks for the timeout it was given, so the exhausted-budget and cancellation cases have to spend the budget to pass; both consumers also pin that an error saying "timed out after 10ms" without the structured detail is not retried. Refs #2422
666283b to
6bf0809
Compare
|
Both review points addressed at 6bf0809 (rebased on 1. The probes now honor the owning request budget. The budget threaded is the one the request already carries into the runner
One clock is created per fingerprint read, so the three probes and their retries
Worst-case wall clock: 45 s, down from ~180 s. A caller with no budget of its 2. Typed timeout classifier in both consumers.
New tests. The immediate-throw mocks are replaced by a fake clock installed
Verified locally against merge-base |
|
The typed timeout check and shared probe budget are improvements, but |
Cause
Two cold-toolchain probes carried per-call budgets sized for a warm toolchain, below the ~18-19s syspolicyd signature-verification stall that blocks the first
xcodebuild/xcrun/large-binary exec after a fresh macOS host boots (the second exec of the same tool is instant):packages/platform-apple/src/snapshot-source/cache-identity.ts:toolOutputranxcodebuild -version,sw_vers,uname,xcrun --show-sdk-versionwithtimeoutMs: Math.min(10_000, remaining).packages/platform-apple/src/runner/runner-cache-metadata.ts:TOOLCHAIN_PROBE_TIMEOUT_MS = 5_000for the runner cache key'sxcodebuild -version/xcrun --show-sdk-version/xcrun --show-sdk-build-versionprobes.Both budgets tripped on cold CI runners even though the surrounding overall deadlines (120s for snapshot-source, the runner preflight budget) had room, producing a toolchain-probe timeout unrelated to the change under test (#2422).
Fix
COLD_TOOLCHAIN_PROBE_TIMEOUT_MS = 30_000, inpackages/platform-apple/src/toolchain-probe-budget.ts, with a comment naming the cold-start stall so the two call sites can't drift apart again.cache-identity.ts'stoolOutputnow bounds each probe byMath.min(COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, remainingSnapshotSourceMs(...))(>= 30s, still bounded by the 120s overall deadline) and retries exactly once after a timeout while the deadline still has room.runner-cache-metadata.ts'srunToolchainProbenow uses the same 30s budget and retries exactly once after a timeout.toolchain-probe-failed/toolchain-probe-emptyon the snapshot-source side, theCOMMAND_FAILED/apple_toolchain_probe_unavailableshape with the existingTOOLCHAIN_PROBE_HINTon the runner side), so existing consumers and tests stay valid.Tests
Added, next to each module's existing tests:
packages/platform-apple/src/snapshot-source/cache-identity.test.ts(new file): a fake host whose first probe call times out and whose second call returns immediately succeeds; a host that always times out still fails with the sameAppErrortimeout text once its single retry is exhausted.packages/platform-apple/src/runner/__tests__/runner-cache-metadata.test.ts: same two cases againstresolveExpectedRunnerCacheMetadata, using device fixtures (IOS_DEVICE,MACOS_DEVICE) untouched by the existing tests so the toolchain fingerprint cache starts empty for each.Verification
vitest runtargeted at both modules, then the fullsnapshot-source/andrunner/suites: all green.pnpm typecheck,pnpm lint,pnpm check:layering: all pass.pnpm check:affected --run: 369 test files / 2589 tests pass.This unblocks the iOS smoke/preflight lane that was failing on cold runners for #2418, #2420, #2421.
Closes #2422