test(coverage): declare every public command's coverage judgments once by thymikee · Pull Request #2418 · callstack/agent-device

thymikee · 2026-09-09T09:50:42Z

Declares every public command's six platform coverage judgments once in
test/integration/command-coverage/declarations.ts, projected per platform at
load time, and deletes the six now-redundant per-platform manifests.

Fixes a review finding: test/integration/command-coverage/ moved the macOS
coverage judgments out from under test/integration/macos-e2e/, but
check-affected's macosCoverageOwnership and isNodeIntegrationPath only
matched the old top-level/macos-e2e/ paths. A declarations.ts edit fell
through to vitest-related, which can't run its actual node --test consumers
(the six coverage smoke tests). Both rules now also match
test/integration/command-coverage/.

Closes #2411.

…judgments once One row per public command in test/integration/command-coverage/declarations.ts carries the android-emulator, ios-simulator, macOS, tvOS, web and Linux classifications with the same fields the six per-platform manifests use today. Each platform's Record<PublicCommand, ...> is projected from that table at load time by a small per-platform view module, so no projected record is committed. No judgment is derived from another platform's: all six stay authored per command.

The six coverage smoke tests, live harnesses and coverage reports now import the platform view module that projects the declaration table. Both the depgraph blast-radius query and the device-lane test follow the iOS and macOS paths.

Their rows now live once, per command, in the declaration table; each platform's record is projected from it at load time.

…ship to command-coverage/ test/integration/command-coverage/declarations.ts now carries the per-command coverage judgments that used to live directly under test/integration/macos-e2e/. Its nested path wasn't matched by macosCoverageOwnership (top-level or macos-e2e/ only) or isNodeIntegrationPath (no nested segments), so it fell through to vitest-related, which can't actually run its node --test consumers. Extend both rules to also match test/integration/command-coverage/.

github-actions · 2026-09-09T09:52:07Z

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.51 MB	4.51 MB	0 B
Package (unpacked)	4.51 MB	4.51 MB	0 B
Package (download)	1.34 MB	1.34 MB	+8 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	23.4 ms	23.3 ms	-0.1 ms
CLI --help	64.4 ms	63.1 ms	-1.3 ms

thymikee · 2026-09-09T10:27:48Z

The projection and affected-test routing look sound at 1b80f01. Please add the migration evidence required by #2411: compare all six projected records with the old manifests, show one historical command addition reduced from six coverage-input files to one, and demonstrate that an omitted declaration fails the exhaustiveness check. The red iOS smoke check appears unrelated: xcrun timed out while querying the Simulator SDK version during installed-bridge preparation.

thymikee · 2026-09-09T11:42:09Z

The iOS smoke lane on this PR is red on the same step in four consecutive runs: Verify clean-installed Simulator snapshot bridge preparation → xcrun timed out after 10000ms inside the toolchain identity probe (packages/platform-apple/src/snapshot-source/cache-identity.ts:87). Every other check is green.

This PR does not touch that path; the same step failed on three unrelated branches in the last three days. Filed as #2422 (cold toolchain probes budgeted below the first-exec signature stall) with a fix PR in progress. Plan: land #2422, rebase, rerun this lane once.

thymikee · 2026-09-09T12:16:09Z

Migration evidence requested above, measured on this branch at 1b80f01e4d against base e7d97f7dfe (full record, unedited):

(1) Projected record vs committed manifest at BASE

Method: the six test/integration/<platform>-e2e/coverage-manifest.ts files were restored
from e7d97f7dfe into the branch worktree alongside the new
test/integration/<platform>-e2e/coverage.ts projections, and both were imported in one
Node process. assert.deepStrictEqual(projected, committed) per platform
(2411/parity-live.mjs, 2411/parity-order.mjs).

platform	committed manifest (BASE)	projected record	result	commands	levels
android-emulator	`android-emulator-e2e/coverage-manifest.ts`	`projectCoverage('androidEmulator')`	equal	54	live 41, command-contract 13
ios-simulator	`ios-simulator-e2e/coverage-manifest.ts`	`projectCoverage('iosSimulator')`	equal	54	live 45, command-contract 8, workflow-live 1
macos	`macos-e2e/coverage-manifest.ts`	`projectCoverage('macos')`	equal	54	live 18, command-contract 21, known-gap 15
tvos	`tvos-e2e/coverage-manifest.ts`	`projectCoverage('tvos')`	equal	54	command-contract 16, known-gap 38
web	`web-e2e/coverage-manifest.ts`	`projectCoverage('web')`	equal	54	live 12, command-contract 41, known-gap 1
linux	`linux-e2e/coverage-manifest.ts`	`projectCoverage('linux')`	equal	54	live 17, command-contract 28, known-gap 9

Every entry is deep-equal, field for field (assertion, evidence reference / owner / scenario,
level, trackingIssue), and no entry's own field order changed
(entriesWithDifferentFieldOrder=0 on all six).

Honest caveat on iteration order: the projected records iterate in the single declaration-table
order. For macos, tvos, web and linux that is the committed manifest's own order. For
android-emulator and ios-simulator the historical authoring order differed, so those two records
now enumerate commands in the shared order. Nothing asserts or persists that order: the smoke
tests compare Object.keys(...).sort(), liveCommandsForScenario(...) feeds
computeMissingCoverage, which filters by evidence counts, and the coverage report is written at
runtime into the artifact dir, not committed.

(2) Planted omission

Removed the whole [C.hover] row (30 lines, all six platform fields) from
test/integration/command-coverage/declarations.ts, changing nothing else.

pnpm typecheck red:
test/integration/command-coverage/declarations.ts(1291,3): error TS2741: Property 'hover' is missing in type '{ artifacts: … }' but required in type 'Record<PublicCommand, CommandCoverageDeclaration>'.
(plus TS6133 for the now-unused Android hover evidence import)
The six coverage smoke tests red at runtime as well: 12 failing of 51, including
"Android emulator coverage exhaustively classifies the public catalog",
"iOS simulator …", "Linux …", "macOS …", "tvOS …", "web coverage exhaustively classifies the
public catalog", and the four report-count tests.
Restored from backup; pnpm typecheck green, 51/51 smoke tests pass, git status --short clean.

(3) Replay of a historical command addition: 6 → 1 coverage input files

Command replayed: hover, added as a public command by
8db36299e4 feat(web): add hover command for hover-gated UI (#1783) (#1786). That commit edited
the two coverage manifests that existed then; at BASE all six manifests classify it and each is
closed by satisfies Record<PublicCommand, …>, so the same addition today costs six.

Measured by deleting hover's coverage rows and counting the files type-checking demands back:

BASE shape (six manifests restored from e7d97f7dfe, hover row stripped from each):
tsc -p tsconfig.json names 6 distinct files, one TS2741 per manifest —
android-emulator-e2e/, ios-simulator-e2e/, linux-e2e/, macos-e2e/, tvos-e2e/,
web-e2e/coverage-manifest.ts.
New shape (declaration removed, step 2 above): 1 file —
test/integration/command-coverage/declarations.ts.

Coverage input files for one public-command addition: 6 → 1. The six judgments, their evidence
references and their gap rows are unchanged; they are authored side by side in one row instead of
in six files.

thymikee · 2026-09-09T12:36:25Z

The added parity, omission and historical replay evidence addresses the remaining review gap at 1b80f01. Ready for human review. The iOS smoke failure is likely unrelated: it times out in the unchanged SDK probe tracked by #2422; whichever of this PR and #2420 lands second must preserve the new hover evidence references.

github-actions · 2026-09-09T13:56:20Z

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-09 13:56 UTC

thymikee added 4 commits September 9, 2026 11:09

test(coverage): delete the six per-platform coverage manifests

aab7f5b

Their rows now live once, per command, in the declaration table; each platform's record is projected from it at load time.

This was referenced Sep 9, 2026

refactor(contracts): additive capability facts pilot on touch family #2420

Open

ios(ci): cold toolchain probes time out below the first-exec signature stall (10 s / 5 s budgets) #2422

Open

thymikee mentioned this pull request Sep 9, 2026

fix(ios): budget cold toolchain probes for the first-exec signature stall #2423

Open

thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 9, 2026

thymikee merged commit 9d7d60c into main Sep 9, 2026
18 of 22 checks passed

thymikee deleted the claude/2409-2411-coverage-declaration branch September 9, 2026 13:55

thymikee mentioned this pull request Sep 9, 2026

refactor: cut the layers and files one feature change must touch (successor to #2278 and #2335) #2409

Open

8 tasks

Navigation Menu

Uh oh!

Uh oh!

test(coverage): declare every public command's coverage judgments once - #2418

test(coverage): declare every public command's coverage judgments once#2418
thymikee merged 4 commits into
mainfrom
claude/2409-2411-coverage-declaration

thymikee commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

thymikee commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Size Report

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

thymikee commented Sep 9, 2026

(1) Projected record vs committed manifest at BASE

(2) Planted omission

(3) Replay of a historical command addition: 6 → 1 coverage input files

Uh oh!

thymikee commented Sep 9, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant