Skip to content

perf(ios): learn generation-scoped native depth hints in the host AX source - #2427

Merged
thymikee merged 1 commit into
claude/agent-device-issue-2424-runner-adapterfrom
claude/agent-device-issue-2424-0284ad
Sep 9, 2026
Merged

perf(ios): learn generation-scoped native depth hints in the host AX source#2427
thymikee merged 1 commit into
claude/agent-device-issue-2424-runner-adapterfrom
claude/agent-device-issue-2424-0284ad

Conversation

@thymikee

@thymikee thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

Third step of #2424, stacked on #2428 (runner adapter) and #2425 (fixture + host adapter). The host Simulator AX source keeps a bounded accepted-depth hint per resolved target id, app generation, and producer. After a recovery that observed a native depth rejection and finished, the next capture of that generation sends nativeLevelsHint, so the guest's first request asks for the accepted levels instead of re-paying the known rejection. A hint changes request strategy only; delivered depth, node bounds, and completeness rules are unchanged. A hint is learned only after the delivered tree validated (a response with sound counters but an unusable tree teaches nothing), never crosses apps, generations, or producers, expires after eight hinted captures (the next one probes the full depth), is never renewed by a hinted success, is left untouched by a failed capture, and explicit raw-depth requests neither use nor teach it. The route's generation circuit stays the only lifecycle owner.

The guest reports its request accounting as recovery (requests, rejected, continuations, accepted levels), emitted as the ios_snapshot_source_recovery diagnostic; a guest that omits it fails closed. Source version v1.5.5. The shared fixture gains the host column of the hint cases and the hinted recovery cases; the hint cases run through the source adapter itself (adapter.test.ts), each step being a real acquisition whose fake guest rejects above the accepted levels and reports boundedness, so rejected, acceptedLevels, and complete are all consumed. The test app gains deep-tree?chain=N&width=M, which reproduces the native depth rejection without a third-party app.

21 files, 875 lines, 343 of them production.

Validation

Tested head: 28a681c570. Runtime sources differ from the benchmarked 2b68a57044 only in the order of learning versus acquisition validation, which changes no request on the benchmarked paths.

  • pnpm check:affected --run --base <#2428 head> passed.
  • Review follow-ups: learning moved after createAcquisition, with a test proving a sound-counters/malformed-tree response leaves the next request unhinted; the fixture's bounded-recovery-still-learned (complete: false, learned) and rejected-at-every-depth-keeps-the-hint (failure, nothing learned) cases now run through the source adapter, so completeness and failure are both exercised.
  • Package growth (+5.2 kB unpacked): depth-hints.ts ~2 kB bundled (the policy itself), adapter.ts + protocol.ts ~1.5 kB (hint decision, recovery parsing, diagnostic), the packaged bridge sources ~2 kB (hint parameter, accounting struct, response block, header docs). A smaller design would drop requests/continuations from the wire and the diagnostic (~0.4 kB) and the 32-target eviction bound (~0.3 kB); both were kept because perf(ios): align AX recovery contracts and evaluate generation-scoped depth hints #2424 requires paired native-call evidence and a bounded owner.
  • Planted red (each restored): hint never applied → 2 fixture cases red; hint leaks across generations → never-across-generations red; hinted success renews → learn-from-complete-recovery red; native levels off by one → 8 red; missing boundary evidence accepted → 7 red; budget 33 → 3 red.
  • Live on an iOS 26.2 simulator, --debug request logs show no-hint → learned, then hinted with zero rejections and one probe-back after eight uses.

Paired benchmark

Base = #2414 head 7c84acfe68; hint = 2b68a57044 (same runtime sources as this head). Source CLI per variant, one dedicated iPhone 17 Pro / iOS 26.2 simulator, isolated state dir per cell, five relaunch rounds (open, first capture, first press, capture) plus ten warm captures per cell, runner warmed once per cell. Only regular captures change (raw explicit-depth requests bypass hints). Noise floor from the ordinary app: ±6 ms on ~55 ms medians. The #2414 Bluesky figure is not attributed here.

screen cell base med / p90 hint med / p90 delta (median)
Settings (ordinary) warm capture, acquire (n=10) 55 / 57 ms 55 / 58 ms +0 ms (+0%)
Settings (ordinary) warm capture, CLI wall (n=10) 258 / 262 ms 250 / 253 ms -8 ms (-3%)
Settings (ordinary) first capture after relaunch, acquire (n=5) 58 / 73 ms 64 / 69 ms +6 ms (+10%)
Settings (ordinary) first press after relaunch, CLI wall (n=5) 1068 / 1090 ms 1058 / 1074 ms -10 ms (-1%)
Settings (ordinary) warm native requests per capture (hint variant) 1.0; 0/10 hinted; 0 rejection(s) (the probe-back) unhinted captures in the same run: 0.0 requests, 0/0 rejected
deep-tree 90×1 (90 nested levels) warm capture, acquire (n=10) 147 / 149 ms 100 / 102 ms -47 ms (-32%)
deep-tree 90×1 (90 nested levels) warm capture, CLI wall (n=10) 353 / 357 ms 296 / 302 ms -57 ms (-16%)
deep-tree 90×1 (90 nested levels) first capture after relaunch, acquire (n=5) 149 / 154 ms 98 / 101 ms -51 ms (-34%)
deep-tree 90×1 (90 nested levels) first press after relaunch, CLI wall (n=5) 1130 / 1174 ms 1063 / 1069 ms -67 ms (-6%)
deep-tree 90×1 (90 nested levels) warm native requests per capture (hint variant) 5.1; 9/10 hinted; 1 rejection(s) (the probe-back) unhinted captures in the same run: 6.0 requests, 5/5 rejected
deep-tree 70×6 (~630 nodes) warm capture, acquire (n=10) 607 / 614 ms 349 / 357 ms -258 ms (-43%)
deep-tree 70×6 (~630 nodes) warm capture, CLI wall (n=10) 814 / 819 ms 554 / 560 ms -260 ms (-32%)
deep-tree 70×6 (~630 nodes) first capture after relaunch, acquire (n=5) 611 / 644 ms 353 / 357 ms -258 ms (-42%)
deep-tree 70×6 (~630 nodes) first press after relaunch, CLI wall (n=5) 1803 / 1822 ms 1548 / 1572 ms -255 ms (-14%)
deep-tree 70×6 (~630 nodes) warm native requests per capture (hint variant) 15.1; 9/10 hinted; 1 rejection(s) (the probe-back) unhinted captures in the same run: 16.0 requests, 5/5 rejected

The ordinary app never consumes a hint. Deep screens drop exactly the rejected request, whose cost grows with tree size (~50 ms at 90×1, ~260 ms at 70×6). After navigating deep → home inside one generation, hinted captures still take one native request with zero continuations (4/4; 63–78 ms vs 77–78 ms base). The probe-back shows as one full-cost capture per eight (warm max 146 / 619 ms, equal to base medians). No cell regressed beyond noise.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.53 MB 4.53 MB +5.3 kB
Package (unpacked) 4.53 MB 4.53 MB +5.3 kB
Package (download) 1.34 MB 1.34 MB +1.5 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.6 ms 27.1 ms -0.5 ms
CLI --help 76.5 ms 74.1 ms -2.3 ms

@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

At 2b68a57, recordRecovery saves the depth hint before createAcquisition validates the returned tree. A response with valid recovery counters but a malformed or over-limit tree can therefore fail acquisition and still affect the next capture. Learn only after acquisition validation succeeds, and add a failed-first-response test proving the next request stays unhinted.

Please also reconcile bounded-recovery learning with #2424: the fixture expects a hint when complete is false, but the test ignores that field. Distinguish a successfully completed bounded request from incomplete recovery, and exercise that distinction rather than leaving an unused completeness claim.

The 5.2 kB unpacked increase needs a short breakdown and the smaller-design tradeoff. Reconcile this branch with the newly split #2425 before merging. The iOS wait failure also occurs on the earlier base, so its relation to this change is unclear; it is not evidence of a clean full E2E run.

@thymikee
thymikee changed the base branch from claude/agent-device-issue-2424-conformance to claude/agent-device-issue-2424-runner-adapter September 9, 2026 14:55
@thymikee
thymikee force-pushed the claude/agent-device-issue-2424-0284ad branch from 2b68a57 to b510ec8 Compare September 9, 2026 14:55
@thymikee
thymikee added this pull request to stack #2426 September 9, 2026 15:02
@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

At b510ec8, recordRecovery still learns before createAcquisition validates the tree, so a failed malformed/over-limit response can influence the next capture. Move learning after successful validation and prove a failed first response leaves the next request unhinted.

The fixture now exercises bounded captures, resolving the unused-field test gap. Its learn-from-bounded policy still needs an explicit reconciliation with #2424’s complete-recovery requirement; bounded is not automatically invalid.

Please itemize the 5.2 kB growth and smaller-design tradeoff. The stack depends on #2428, which currently conflicts with #2425; iOS CI remains pending.

…source

The host source remembers the native levels a finished recovery accepted,
keyed by resolved target id, app generation, and producer, and sends them as
nativeLevelsHint so the guest's first request skips the known rejection. The
hint changes request strategy only: delivered depth, node bounds, and
completeness rules are unchanged. It is learned only after the delivered tree
validated, expires after eight hinted captures, is never renewed by a hinted
success, and never crosses apps, generations, or producers; explicit raw-depth
requests neither use nor teach it. The route's generation circuit stays the
only lifecycle owner.

The guest reports its request accounting (requests, rejected, continuations,
accepted levels) as recovery, which the host emits as the
ios_snapshot_source_recovery diagnostic; the source version moves to v1.5.5.
The shared recovery fixture gains the host column of every hint case and the
hinted recovery cases, replayed through the source adapter itself. The test
app gains a deep-tree screen that reproduces the native depth rejection for
paired benchmarks.
@thymikee
thymikee force-pushed the claude/agent-device-issue-2424-0284ad branch from b510ec8 to 28a681c Compare September 9, 2026 15:34
@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Addressed at 28a681c570, rebased onto #2428#2425.

  • Learning now runs after createAcquisition validates the delivered tree. New test: a response with sound recovery counters but a malformed tree fails acquisition, emits no recovery diagnostic, and the next request of the same generation is unhinted.
  • Completeness versus failure is exercised, not declared: the fixture's hint cases run through the source adapter (adapter.test.ts), each step a real acquisition whose fake guest rejects above the accepted levels and reports boundedness. bounded-recovery-still-learned (complete: false → learned, mirroring the runner's policy) and the new rejected-at-every-depth-keeps-the-hint (halving budget exhausted → acquisition fails → nothing learned, memory untouched) both execute on both producers; test(ios): run the shared AX recovery fixture through the runner and characterize its depth memory #2428 carries the runner column.
  • Size: +5.2 kB unpacked = depth-hints.ts ~2 kB bundled, adapter.ts/protocol.ts ~1.5 kB, packaged bridge sources ~2 kB. The smaller design would drop requests/continuations from the wire and diagnostic (~0.4 kB) and the 32-target eviction bound (~0.3 kB); kept because perf(ios): align AX recovery contracts and evaluate generation-scoped depth hints #2424 asks for paired native-call evidence and a bounded owner. Details in the body.
  • The runtime change since the benchmarked head is only the learn/validate ordering, which issues no different request on the benchmarked paths; the table stands.

@thymikee

thymikee commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

The validation-before-learning fix at 28a681c addresses the previous bug, and the package-growth breakdown explains the retained state and diagnostics. No remaining runtime blocker found. One wording nit: failed hinted requests still consume an expiry use, so describe them as leaving the learned depth unchanged, not all hint state untouched. Merge #2414, #2425 and #2428 first; checks on this head are still running.

@thymikee
thymikee merged commit 52420d7 into main Sep 9, 2026
23 of 24 checks passed
@thymikee
thymikee deleted the claude/agent-device-issue-2424-0284ad branch September 9, 2026 16:24
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-09 16:24 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant