iOS: first-class widget and Live Activity testing through a system UI target

[thymikee]

iOS: first-class widget and Live Activity testing through a system UI target

Motivation

Agent Device can launch an app, press Home, and take a full-screen screenshot, but its semantic snapshot and interaction route remains bound to the session app. That prevents an agent from reliably installing, inspecting, and editing widgets on SpringBoard, and from observing Live Activities outside the app.

The goal is first-class support for testing widget and Live Activity experiences while preserving the distinction between:

• raw capability: target iOS system UI for snapshots, selectors, and interactions;
• workflow guidance: explain how an agent can add/edit a widget using the UI exposed by the current iOS version;
• visual verification: use screenshots for surfaces such as Dynamic Island where semantic AX coverage can be sparse.

Architecture decision

Use the existing XCTest runner and XCUIApplication(bundleIdentifier: "com.apple.springboard") as the production system-UI backend.

Do not add idb as a runtime dependency and do not make idb/CoreSimulator private APIs the contract:

• idb's ui describe-all is explicitly simulator-only;
• its current implementation calls CoreSimulator's private sendAccessibilityRequestAsync API and AccessibilityPlatformTranslation/AXPTranslator;
• that is a useful prototype oracle for SpringBoard hierarchy quality, but it cannot cover physical devices and adds another Xcode-private compatibility surface.

Agent Device already has a narrower private XCTest AX snapshot bridge in RunnerAXSnapshotBridge.m. Keep that path simulator-only and fallback-only. If SpringBoard's normal XCTest snapshot is slow or fails, adapt the existing bridge to the SpringBoard XCUIApplication; do not introduce idb or a second host-side hierarchy implementation.

Production priority:

1. Public XCTest APIs against SpringBoard on simulator and physical device.
2. Existing private AX bridge as a bounded simulator snapshot fallback.
3. Screenshot plus coordinate interaction when the system surface does not expose useful accessibility nodes.

Proposed command contract

Add a request-scoped automation target rather than changing session ownership:

• CLI: --ui-target app|system
• Node/MCP input: uiTarget: "app" | "system"
• default: app

Do not reuse --target; it already describes the device family (mobile, tv, desktop). Do not store system as the session app bundle. The app session must remain intact while individual commands target SpringBoard.

Apply the option to semantic and interactive commands that need a UI root:

• snapshot, find, get, is, wait
• click, fill, press, longpress, scroll, gesture

Full-screen iOS screenshots already observe the display; they do not need a different capture backend. They should still compose correctly in a system-target workflow.

For uiTarget=system:

• resolve the runner application/root to com.apple.springboard;
• do not activate or relaunch the session app during command preflight;
• preserve the existing session app bundle and lifecycle;
• expose target provenance once per response, not once per node;
• bind snapshot refs to their UI target and reject an app ref used by a system command (and vice versa) with a typed, actionable error;
• include the target in recording/replay identity where needed so replay cannot silently cross roots.

Do not add a widget add command. Widget installation/editing remains ordinary system UI automation.

Prototype gate

Before broad TypeScript plumbing, prove the runner path on the configured iOS simulator and a physical iPhone when available.

1. Open the fixture app, press Home, and capture SpringBoard through normal XCTest.
2. Long-press empty Home Screen space and capture edit mode.
3. Verify the hierarchy exposes enough current, localized labels/roles to discover the controls corresponding to Edit, Add Widget, widget search/results, Add Widget, and Done.
4. Compare the simulator hierarchy with:
   • the existing private AX bridge adapted temporarily to SpringBoard;
   • idb ui describe-all as a diagnostic oracle only, when idb is already available.
5. Record node counts, elapsed time, missing controls, and screenshots. Confirm normal XCTest is viable as the primary path.
6. On a physical iPhone, prove at minimum a SpringBoard snapshot and one selector-driven interaction. Private AX/idb are not acceptable evidence there.

If normal XCTest cannot produce a useful SpringBoard hierarchy on physical iOS, stop and report the capability boundary before implementing a simulator-only feature under a cross-device contract.

Implementation shape

Keep the change layered and deliver it as stacked, reviewable PRs.

PR 1: system UI target and provenance

• Add uiTarget to the central command descriptors/flag groups and only the command families listed above.
• Project it through CLI, Node/MCP contracts, daemon request context, and Apple runner command models.
• Resolve app to the session appBundleId and system to the runner's existing SpringBoard application.
• Separate app-activation preflight from root selection so system commands never reactivate the app.
• Add response-level target metadata and ref-frame target provenance.
• Reject cross-target refs deterministically.
• Classify affected interaction guarantee cells under ADR-0011 and add real interaction-contract scenarios.
• Preserve macOS surface behavior and Android behavior; this feature is iOS-only initially.

Likely ownership areas:

• src/contracts/cli-flags.ts
• src/commands/cli-grammar/flag-definitions-target.ts and flag groups
• owning command-family contracts/projections
• daemon dispatch context and Apple request projection
• src/platforms/apple/core/runner/runner-contract.ts
• Swift runner request models and command execution/root selection
• snapshot/ref-frame store and replay provenance

Do not grow RunnerTests+CommandExecution.swift into another mixed-responsibility file. Extract a focused application/root resolver if the change would otherwise add branching throughout command execution.

PR 2: deterministic widget/Live Activity fixture

Extend the repo-owned React Native test app with a minimal Voltra-backed fixture:

• one clearly named widget with deterministic text;
• one editable/configurable state if Voltra supports it without remote services;
• one interactive widget action where the deployment target permits it;
• one Live Activity with deterministic start, update, and end controls;
• compact/minimal/expanded Dynamic Island presentations and a lock-screen presentation;
• App Group/configuration needed by the extension;
• local ActivityKit start/update/end first; APNs remote updates are a later physical-device extension.

Keep fixture UI compact and purpose-built for automation. Do not add authored Objective-C merely to host Voltra. Track generated/build dependency size separately from authored source size.

Expected order-of-magnitude impact to verify in the PR, not assume:

• authored RN/config code: roughly 300-700 lines;
• authored Swift/Objective-C for the fixture: ideally zero or a small generated/config shim;
• Voltra dependency checkout: a few MB;
• app/extension binary and build-time delta: measure before/after and report in the PR body.

PR 3: help and live evidence

Add an iOS system-surfaces help topic and focused SkillGym coverage. Guidance should describe the workflow without hard-coding English labels as protocol:

1. open the app and prepare widget/Live Activity state;
2. press Home;
3. use --ui-target system snapshot -i to inspect the current localized SpringBoard controls;
4. long-press an empty coordinate to enter edit mode;
5. use selectors from the fresh snapshot to open the widget gallery, search, select a size, add, and finish;
6. long-press an installed widget and inspect the fresh hierarchy to edit it;
7. use screenshots for visual assertions and as the fallback when AX is sparse;
8. start/update/end Live Activities in the app, then inspect SpringBoard/lock-screen semantics where available and capture screenshots for Dynamic Island.

The help must say that system labels vary by iOS version and locale. Agents should discover them from the current snapshot instead of relying on literal Edit/Done strings.

Acceptance criteria

• An app session remains open while app and system commands alternate; switching roots does not relaunch or lose app state.
• press home followed by snapshot --ui-target system -i returns a useful SpringBoard tree on a supported simulator and physical iPhone.
• An agent can add the deterministic fixture widget using a coordinate long-press plus selector-driven system interactions discovered from snapshots.
• An agent can long-press the installed widget and enter its edit/configuration path when the widget exposes one.
• App refs and system refs are not interchangeable; misuse returns a typed error and recovery hint.
• Snapshot, selector, recording/replay, timeout, warnings, and diagnostics behavior remains consistent with existing contracts.
• Live Activity start/update/end is deterministic from the fixture app.
• Lock-screen and Dynamic Island screenshots demonstrate each supported fixture state. Semantic assertions are required only where the OS exposes stable AX nodes.
• Live evidence includes simulator and physical-device results, exact OS/Xcode/device identifiers, commands, screenshots/artifacts, and any AX gaps.
• Required gates pass: affected checks, provider integration/coverage for response changes, interaction contracts, pnpm build:xcuitest, formatting, and live session cleanup.

Non-goals

• No direct WidgetKit database manipulation or private API to install a widget.
• No widget-specific orchestration command.
• No hard-coded English SpringBoard workflow.
• No idb dependency or host-side clone of idb's accessibility serializer.
• No promise that Dynamic Island content is fully semantic; visual evidence is first-class there.
• No lock/unlock/passcode automation contract in the first slice.
• No APNs server or remote push setup in the first slice.

Risks to report, not hide

• SpringBoard AX shape and labels vary across iOS/Xcode versions and locale.
• Private AX fallback is simulator-only and Xcode-private; selector availability must be feature-detected and failure must fall back cleanly.
• System UI may expose controls only after animations settle; use existing settle/timeout semantics rather than fixed sleeps.
• Empty-space long-press is inherently coordinate-based; document it as the one deliberate coordinate step.
• Widget extension packaging can inflate checkout/build/app size; measure actual deltas before merge.

Source references for the architecture decision

• idb accessibility docs: website/docs/accessibility.mdx states these primitives are for iOS Simulators.
• idb implementation: FBSimulatorAccessibilityCommands.swift imports AccessibilityPlatformTranslation and calls CoreSimulator's private accessibility request selector.
• Agent Device existing fallback: RunnerAXSnapshotBridge.m uses XCTest's private accessibility interface and is compiled only for the iOS simulator by RunnerTests+AXSnapshotFallback.swift.
• Agent Device already owns a SpringBoard XCUIApplication in RunnerTests.swift; extend that application/root seam instead of adding another backend.