Skip to content

ios(ci): cold toolchain probes time out below the first-exec signature stall (10 s / 5 s budgets) #2422

Description

@thymikee

Symptom

The iOS smoke lane fails on cold macOS runners with a toolchain probe timeout, unrelated to the PR under test:

  • Verify clean-installed Simulator snapshot bridge preparation: AppError: xcrun timed out after 10000ms (from verify-installed-snapshot-bridge.tsensureSnapshotBridgeBinary).
  • Preflight iOS runner through public CLI: Could not read the Xcode toolchain versions the Apple runner cache is keyed on (xcodebuild -version: … timed out after 5000ms).

Occurrences in the last three days, all on the same step and with no code in the failing path changed by the branch: refactor/core-replay-divergence-* (2026-09-07 16:06), claude/2339-contracts-single-owner-moves (09-06 10:23), claude/wait-poll-timeline (09-06 09:07), PR #2418 three consecutive runs on 2026-09-09 (09:50, 10:43 preflight variant, 10:51), PR #2420 (10:49, 45 s variant). Six other branches' iOS runs passed in the same window, so it is per-runner cold state, not a fleet outage.

Cause

Two cold-toolchain probes carry budgets below the first-exec stall of Apple's signature verification on a fresh host (the first xcodebuild/xcrun/large-binary exec after boot blocks roughly 18 to 19 s at 0% CPU while syspolicyd scans; the second call is instant):

  • packages/platform-apple/src/snapshot-source/cache-identity.ts:87: toolOutput runs xcodebuild -version, sw_vers, uname, xcrun --sdk iphonesimulator --show-sdk-version with timeoutMs: Math.min(10_000, remaining), under a 120 s overall deadline set by the verify script.
  • packages/platform-apple/src/runner/runner-cache-metadata.ts:27: TOOLCHAIN_PROBE_TIMEOUT_MS = 5_000 for xcodebuild -version in the runner cache key.

Both budgets were sized for a warm toolchain. The overall deadlines (120 s snapshot-source, the runner preflight budget) have room; only the per-call budget trips.

Required behavior

  • A toolchain identity probe tolerates one cold-start stall: per-call budget of at least 30 s bounded by the remaining deadline, and one retry after a timeout while the deadline allows, because the retry is instant once the signature check has run.
  • Both sites read the same constant and carry one comment naming the stall, so the two do not drift again.
  • The timeout error keeps its current text and code so existing tests and the runner-cache hint stay valid.
  • Unit tests: a fake host whose first probe call exceeds the budget and whose second returns immediately passes; a host that never returns still fails at the deadline with the same error.

Non-goals

  • Raising the snapshot-bridge native build timeout (BUILD_TIMEOUT_MS) or the overall deadlines.
  • Retrying the compile step.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions