Skip to content

fix(ios): budget cold toolchain probes for the first-exec signature stall - #2423

Open
thymikee wants to merge 3 commits into
mainfrom
claude/2422-cold-toolchain-probe
Open

fix(ios): budget cold toolchain probes for the first-exec signature stall#2423
thymikee wants to merge 3 commits into
mainfrom
claude/2422-cold-toolchain-probe

Conversation

@thymikee

@thymikee thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member

Cause

Two cold-toolchain probes carried per-call budgets sized for a warm toolchain, below the ~18-19s syspolicyd signature-verification stall that blocks the first xcodebuild/xcrun/large-binary exec after a fresh macOS host boots (the second exec of the same tool is instant):

  • packages/platform-apple/src/snapshot-source/cache-identity.ts: toolOutput ran xcodebuild -version, sw_vers, uname, xcrun --show-sdk-version with timeoutMs: Math.min(10_000, remaining).
  • packages/platform-apple/src/runner/runner-cache-metadata.ts: TOOLCHAIN_PROBE_TIMEOUT_MS = 5_000 for the runner cache key's xcodebuild -version / xcrun --show-sdk-version / xcrun --show-sdk-build-version probes.

Both budgets tripped on cold CI runners even though the surrounding overall deadlines (120s for snapshot-source, the runner preflight budget) had room, producing a toolchain-probe timeout unrelated to the change under test (#2422).

Fix

  • Added one shared constant, COLD_TOOLCHAIN_PROBE_TIMEOUT_MS = 30_000, in packages/platform-apple/src/toolchain-probe-budget.ts, with a comment naming the cold-start stall so the two call sites can't drift apart again.
  • cache-identity.ts's toolOutput now bounds each probe by Math.min(COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, remainingSnapshotSourceMs(...)) (>= 30s, still bounded by the 120s overall deadline) and retries exactly once after a timeout while the deadline still has room.
  • runner-cache-metadata.ts's runToolchainProbe now uses the same 30s budget and retries exactly once after a timeout.
  • Error codes and messages are unchanged in both places (toolchain-probe-failed/toolchain-probe-empty on the snapshot-source side, the COMMAND_FAILED / apple_toolchain_probe_unavailable shape with the existing TOOLCHAIN_PROBE_HINT on the runner side), so existing consumers and tests stay valid.

Tests

Added, next to each module's existing tests:

  • packages/platform-apple/src/snapshot-source/cache-identity.test.ts (new file): a fake host whose first probe call times out and whose second call returns immediately succeeds; a host that always times out still fails with the same AppError timeout text once its single retry is exhausted.
  • packages/platform-apple/src/runner/__tests__/runner-cache-metadata.test.ts: same two cases against resolveExpectedRunnerCacheMetadata, using device fixtures (IOS_DEVICE, MACOS_DEVICE) untouched by the existing tests so the toolchain fingerprint cache starts empty for each.

Verification

  • vitest run targeted at both modules, then the full snapshot-source/ and runner/ suites: all green.
  • pnpm typecheck, pnpm lint, pnpm check:layering: all pass.
  • pnpm check:affected --run: 369 test files / 2589 tests pass.
  • No simulator/device runs (not needed for this fix; the change is exec-timeout budgeting only).

This unblocks the iOS smoke/preflight lane that was failing on cold runners for #2418, #2420, #2421.

Closes #2422

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.51 MB 4.51 MB +1.3 kB
Package (unpacked) 4.51 MB 4.51 MB +1.3 kB
Package (download) 1.34 MB 1.34 MB +380 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.2 ms 29.8 ms +1.5 ms
CLI --help 78.7 ms 79.8 ms +1.1 ms

@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Fixed the eager-closure-budgets failure (666283b).

Root cause: runner-cache-metadata.ts sits in the eager closure of all seven
gated entries, so its new import of toolchain-probe-budget.ts added one
module to each. snapshot-source/cache-identity.ts, the other consumer, was
never actually reached by any of the seven closures — its own import didn't
matter for the gate.

Fix: deleted toolchain-probe-budget.ts. COLD_TOOLCHAIN_PROBE_TIMEOUT_MS
now lives inline in snapshot-source/cache-identity.ts (the file it was
already only reachable through). runner-cache-metadata.ts declares its own
copy of the same constant instead of importing it — no shared module means
no new module in any of the seven closures. A new unit test in
runner/__tests__/runner-cache-metadata.test.ts imports both constants and
asserts they're equal, so the two copies can't drift apart.

Verified locally against merge-base e7d97f7dfe (origin/main):

  • eager-closure-budgets.test.ts: 577/577 pass, all seven entries back to
    their merge-base counts (120/61/101/60/47/100/13), no APPROVED_OVER_CEILING
    row added.
  • check:layering: OK (220/220).
  • typecheck and lint: clean.
  • cache-identity.test.ts and runner-cache-metadata.test.ts (including the
    two cold-start-retry tests this PR added): all pass.

@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

The runner probes need to honor the remaining request budget at 666283b. Three synchronous probes can each run twice for 30 seconds, blocking for roughly 180 seconds without a deadline or cancellation check. Please pass the owning budget through, cap each attempt by its remaining time, and test exhausted and canceled requests; the current immediate-throw mocks do not prove the deadline.

Both retry classifiers also inspect error text. The subprocess timeout already carries structured timeout details. Please classify that signal instead, and test that a non-timeout error with similar wording is not retried.

…tall

xcodebuild/xcrun toolchain probes in cache-identity.ts and
runner-cache-metadata.ts were budgeted for a warm toolchain (10s/5s),
below the ~18-19s syspolicyd signature-verification stall on the first
exec after a fresh macOS host boots. Share one 30s floor constant
between both call sites and retry once after a timeout while the
deadline allows, since the second exec is instant.

Closes #2422
…ared module

toolchain-probe-budget.ts sat outside every platform-apple facade's eager
closure, but runner-cache-metadata.ts (imported from it) sits inside all
seven -- so the new import added one module to each, tripping the
eager-closure-budgets no-growth gate (#2422).

Delete the shared module. cache-identity.ts keeps the canonical constant
inline (it was already outside the gated closures); runner-cache-metadata.ts
declares its own copy, guarded by a new unit test that asserts the two
stay equal.
Three synchronous probes could each retry once at 30 s, so a wedged
toolchain host blocked a request for ~180 s with no deadline and no
cancellation check.

The runner cache decision now takes the owning request's budget
(remaining ms + abort signal) and builds one clock per fingerprint read:
every attempt runs at min(per-call ceiling, remaining), the retry is
skipped once the budget is spent, an exhausted budget fails the decision
without starting another probe, and an aborted signal surfaces the
cancellation instead of retrying. `ensureXctestrunArtifact` passes the
build budget and signal, session reuse passes the startup budget and the
request signal, and lease adoption passes the startup budget; a caller
with neither is still capped at 45 s total, so the worst case falls from
~180 s to 45 s. Error codes, texts, and the probe hint are unchanged.

Both retry classifiers now read the exec layer's structured timeout
detail instead of matching "timed out after Nms" in the message. The
predicate is exported once from host-kit's command surface and reaches
`runner-cache-metadata.ts` through the Apple runner host port, so the
file's eager closure is unchanged.

Tests use a fake clock that only advances when a probe actually blocks
for the timeout it was given, so the exhausted-budget and cancellation
cases have to spend the budget to pass; both consumers also pin that an
error saying "timed out after 10ms" without the structured detail is not
retried.

Refs #2422
@thymikee
thymikee force-pushed the claude/2422-cold-toolchain-probe branch from 666283b to 6bf0809 Compare September 9, 2026 14:56
@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Both review points addressed at 6bf0809 (rebased on 9d7d60c5e0).

1. The probes now honor the owning request budget.

The budget threaded is the one the request already carries into the runner
preflight — there is no ambient deadline below it, so it is passed explicitly as
RunnerCacheProbeBudget { timeoutMs?, signal? }, a third argument to
resolveExpectedRunnerCacheMetadata:

  • ensureXctestrunArtifactoptions.buildTimeoutMs + options.signal. That
    is the prepareDeadline-derived phase timeout from fix(ios): honor the startup budget through a cold Simulator boot #2325
    (readPrepareDeadlinePhaseTimeouts sets buildTimeoutMs from
    prepareDeadline.remainingMs()), and the same abort signal the build itself
    gets.
  • ensureRunnerSession's reuse path → options.startupTimeoutMs +
    resolveRunnerRequestSignal(options).
  • tryAdoptRunnerSessionFromLeaseoptions.startupTimeoutMs.

One clock is created per fingerprint read, so the three probes and their retries
share a single budget rather than each getting its own:

  • every attempt runs at min(COLD_TOOLCHAIN_PROBE_TIMEOUT_MS, remaining);
  • no retry once the remaining time is 0 — the original timeout error propagates;
  • a probe reached with nothing left returns a probe_error failure instead of
    starting another blocking spawnSync;
  • an aborted signal throws the request-canceled error (checked before each probe
    and after each failed attempt), so cancellation surfaces instead of being
    folded into a toolchain failure. spawnSync cannot be interrupted mid-call, so
    cancellation is observed between attempts — the per-attempt cap is what bounds
    how long that takes; this is stated on the type.

Worst-case wall clock: 45 s, down from ~180 s. A caller with no budget of its
own is capped by TOOLCHAIN_FINGERPRINT_BUDGET_MS = 45_000 (one 30 s stall plus
its now-warm retry and the two remaining probes); a caller with less time is
capped by its own. Error codes, texts, retriable, and TOOLCHAIN_PROBE_HINT
are unchanged — an exhausted budget still fails as
COMMAND_FAILED / apple_toolchain_probe_unavailable with the hint.

2. Typed timeout classifier in both consumers.

isCommandTimeoutError(error) is now exported once from
packages/host-kit/src/internal/exec.ts via @agent-device/host-kit/command:
AppError with code COMMAND_FAILED and a numeric details.timeoutMs, the
detail both createTimeoutError and the runCmdSync ETIMEDOUT branch stamp.
snapshot-source/cache-identity.ts imports it directly. runner-cache-metadata.ts
reaches it through the Apple runner host port (runner/host.ts +
core/runner-host.ts) rather than a new import, so its eager closure is
unchanged. Both local regex helpers are deleted.

New tests. The immediate-throw mocks are replaced by a fake clock installed
through the test host's deadlineFromTimeoutMs, advanced only by a fake exec
that blocks for the whole timeout it was handed — a case claiming the budget was
spent has to spend it.

runner/__tests__/runner-cache-metadata.test.ts (describe('toolchain probe budget'),
each case starting from an empty fingerprint memo):

  • cold-start recovery: attempt timeouts are exactly [30000, 15000] — the retry
    runs on what the shared budget has left, not a fresh ceiling;
  • never-returning host: 2 execs total and clock.nowMs === 45_000; the two xcrun
    probes fail on the budget instead of blocking 30 s each;
  • owning request with 4 s left: one 4 s attempt, no retry, clock.nowMs === 4_000;
  • canceled while a probe blocked: 1 exec, request-canceled error surfaces;
  • already-canceled request: 0 execs;
  • COMMAND_FAILED saying timed out after 10ms with no timeoutMs detail: 1
    attempt, not retried.

snapshot-source/cache-identity.test.ts: the same non-timeout look-alike is not
retried (1 call), and the always-timing-out case now pins exactly 2 calls.
host-kit/src/internal/exec.test.ts: the predicate is true for real runCmd and
runCmdSync timeouts and false for the look-alike message, a plain Error, and
undefined.

Verified locally against merge-base 9d7d60c5e0: eager-closure-budgets.test.ts
581/581 (no growth on any of the seven entries), typecheck, lint,
check:layering (220/220), check:affected --run (715 files / 5351 tests), and
the apple-runner project suite (382 tests). No device runs.

@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

The typed timeout check and shared probe budget are improvements, but ensureXctestrunArtifact gives probing options.buildTimeoutMs and then gives the build that same full timeout again. A cold probe can add 30–45 s on top of the phase budget. Pass one remaining deadline through both and add a probe-plus-build regression. The snapshot-source timeout test also needs to advance its clock and prove the retry receives only the remainder. Finally, #2422 asks both consumers to read one timeout constant: move it to a small dependency-free owning module instead of keeping two copies plus an equality test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ios(ci): cold toolchain probes time out below the first-exec signature stall (10 s / 5 s budgets)

1 participant