[skills] Update CI status skill for public pipeline by simonrozsival · Pull Request #11670 · dotnet/android

simonrozsival · 2026-06-16T18:52:54Z

Summary

Adapts the local agent CI guidance to the current dotnet/android PR validation pipeline. Docs/skills only — no product code.

PR validation runs on the public dotnet-android Azure DevOps pipeline on dev.azure.com/dnceng-public (project public, definition 333) for every PR.

Changes

Rework the ci-status skill around the single public dotnet-android build.
Extract build IDs from GitHub check links and query dnceng-public/public directly.
Handle fork/direct PRs as trigger/approval differences, not different pipeline selections.
Use az rest + ResultsByBuild for failed-test queries because the normal AZDO test runs API path is broken on dnceng-public.
Add failure summaries, cross-config failed-test matrix reporting, crash/incomplete-lane detection, and clearer gating-vs-flaky verdicts.
Update .github/copilot-instructions.md for the public pipeline.

Validation

Exercised the updated ci-status skill end-to-end against live PRs and builds:

In-progress build with a gating failure ([gh-aw] disable AI-credit guardrails (max-daily / max-ai) #11659) — flagged the failed package-test stage and failed tests.
Green build with non-blocking flaky device-test failures ([Mono.Android] Remove dead Java.Interop.__TypeRegistrations plumbing #11667) — reported green from build/check status while surfacing the failures as non-gating.
Fully red build ([NativeAOT] Remove managed NativeAOT type map in favor of the trimmable type map #11643) — reported red checks and failed tests.

All az, gh, and az rest command paths used by the skill were run against dnceng-public and confirmed working.

Adapt the local agent skills and workflow docs to two recent CI changes: - dotnet#11578 made dnceng-public `dotnet-android` the single PR pipeline (full matrix for every PR) and set `pr: none` on the DevDiv `Xamarin.Android-PR` pipeline. Rework the `ci-status` skill around one public build: build-id extraction, AZDO test queries via `az rest` (ResultsByBuild), queue-driven ETA variance, a gating-vs-flaky verdict (build result + checks are authoritative; `continueOnError` device-test failures are non-gating), and device-test `logcat-*.txt` artifacts. - dotnet#11224 migrated Mono.Android device tests from NUnitLite/`-t:RunTestApp` to stock NUnit via `dotnet test` + Microsoft Testing Platform. Update the `tests` skill, `copilot-instructions.md`, and `UnitTests.md` to the new `-t:Install` + `dotnet test --no-build --report-trx` flow (TRX output), and drop the removed NUnitLite row from the `update-tpn` skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Apply the skill-creator "concise is key" principle to the ci-status skill: cut content that is obvious to the agent and keep only what is unique to dotnet/android CI. - SKILL.md 306 -> 78 lines (~70% smaller always-loaded body): drop the prerequisites table, PowerShell duplicates, step-by-step arithmetic (elapsed/median/job-counter explanations) and the verbose output template; state the repo-specific facts once (single public pipeline, Xamarin.Android-PR pr:none, test-area 404 -> az rest, gating-vs-flaky continueOnError lanes, queue-time ETA variance, logcat artifact). - Move deep-dive commands (ETA query, per-test error/stack, test-runs list, log fetch) to references/azdo-queries.md, loaded only when needed. Validated with skill-creator quick_validate and by re-running the lean workflow against live PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per review feedback: the explicit "Present summary" template produced predictable, familiar output. Bring back an explicit (but lean) report template — PR header + fork badge, the dotnet-android build block (result / elapsed / ETA / job counters), a Stage > Job status table, Failures, gating-vs-flaky Failed tests, Verdict, and What next — in place of the prose-only guidance from the previous commit. SKILL.md is still ~107 lines (vs 306 originally). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Expand the ci-status skill's first summary (Phase 1, Step 3d) with a new scripts/ci_failures.py that turns raw build failures into: - a per-test cross-config matrix: for each test that failed in >=1 config, which flavors/OSes it failed vs passed in, with same-build retries shown as "Failed->Passed (retry)" (a retry that passes => flaky), plus the assembly and the assert/stack trace - crashed / incomplete lane detection: lanes that went red with no usable failed-test list ("Zero tests ran" startup crash, incomplete run, or timeout/hang), excluding normal failed-test lanes; points at the device logcat for the started-but-never-finished culprit - branch cross-reference: PR-changed files whose name matches a failing test's class/namespace/assembly Also document the crash-culprit-from-logcat recipe in references/azdo-queries.md and fix the error-patterns.md job-timeout signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove duplicated tests skill and workflow documentation updates from this PR, leaving the focused ci-status changes and refreshed Copilot guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the MTP device-test reproduction snippet from this CI-status PR; that documentation now lives in the tests-skill PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep CI guidance focused on the public dotnet-android pipeline without mentioning the retired PR pipeline path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Updates the repository’s Copilot “ci-status” skill and related references to match the current dotnet/android PR validation setup: a single public Azure DevOps dotnet-android pipeline (dnceng-public/public/333) whose jobs surface as GitHub checks.

Changes:

Rewrites .github/skills/ci-status/SKILL.md to triage the single public pipeline and adds a Phase 2 “deep dive” path via a bundled script.
Adds ci_failures.py to generate a cross-config failed-test matrix, detect crash/incomplete lanes, and optionally cross-reference PR-changed filenames.
Updates supporting references and top-level Copilot instructions to align guidance with the public pipeline.

Show a summary per file

File	Description
.github/skills/ci-status/SKILL.md	Reworked ci-status procedure around a single dnceng-public build and added deep-dive workflow.
.github/skills/ci-status/scripts/ci_failures.py	New helper script for richer failure analysis (matrix/crash detection/xref).
.github/skills/ci-status/references/error-patterns.md	Refines timeout detection pattern guidance.
.github/skills/ci-status/references/binlog-analysis.md	Updates artifact naming expectations for public pipeline outputs.
.github/skills/ci-status/references/azdo-queries.md	New AZDO query reference for dnceng-public endpoints and workflows.
.github/skills/android-reviewer/SKILL.md	Removes outdated guidance about internal-only Xamarin.Android-PR check gating.
.github/copilot-instructions.md	Updates CI investigation guidance to reference the public `dotnet-android` pipeline + ci-status skill.

Copilot's findings

Files reviewed: 7/7 changed files
Comments generated: 4

- SKILL.md / azdo-queries.md: scope the 404 claim to the `--area test --resource runs` endpoint (other test resources, e.g. `--resource results`, work fine on dnceng-public). - SKILL.md: call out the required `azure-devops` az CLI extension alongside `gh`/`az`. - ci_failures.py: stop printing a hardcoded "Test Results - APKs .NET Debug - macOS 1" artifact name. Print an `artifact list` command filtered to `Test Results - *` and a placeholder so the user picks the artifact for the actual failing lane. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jonathanpeppers · 2026-06-17T22:12:11Z

+#!/usr/bin/env python3
+"""Enriched failure analysis for one dnceng-public `dotnet-android` build:
+  1. cross-config matrix per failed test (failed/passed/retried configs) + stack/asserts
+  2. crashed / incomplete lanes (started-but-not-finished culprit lives in logcat)
+  3. branch cross-reference (PR changes that name a failing test's class/namespace/assembly)
+
+Needs `az login`. Usage: ci_failures.py --build-id N [--pr N] [--repo dotnet/android]
+"""


I'm fine to leave this, but should we port it to dotnet run ci_failures.cs, so it is C#?

@copilot this is good feedback, rewrite this script from Python to C# (file-based app)

simonrozsival · 2026-06-26T10:58:35Z

@copilot migrate .github/skills/ci-status/scripts/ci_failures.py from python to C# (.github/skills/ci-status/scripts/ci_failures.cs) and update the skill to properly run this C# file-based app.

Rewrite the ci-status deep-analysis helper from Python to a C# file-based app (ci_failures.cs), run via `dotnet run ci_failures.cs`, per PR review feedback. Update SKILL.md and references/azdo-queries.md to invoke and reference the C# app. Also resolves two review comments while porting: the crash section now prints a placeholder artifact name plus an `az pipelines runs artifact list` hint instead of a hard-coded lane, and the logcat grep uses a valid single-backslash line continuation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

simonrozsival · 2026-06-29T14:51:00Z

CI failure is unrelated to this PR

simonrozsival and others added 5 commits June 16, 2026 20:50

Scope CI skill update to CI changes

88f4f04

Remove duplicated tests skill and workflow documentation updates from this PR, leaving the focused ci-status changes and refreshed Copilot guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

simonrozsival changed the title ~~[skills] Update CI/test skills for public pipeline + NUnit dotnet test~~ [skills] Update CI status skill for public pipeline Jun 17, 2026

simonrozsival and others added 2 commits June 17, 2026 11:41

Keep device test guidance in tests PR

56318dd

Remove the MTP device-test reproduction snippet from this CI-status PR; that documentation now lives in the tests-skill PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove obsolete pipeline references from agent docs

7ce3227

Keep CI guidance focused on the public dotnet-android pipeline without mentioning the retired PR pipeline path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

simonrozsival marked this pull request as ready for review June 17, 2026 09:57

simonrozsival requested a review from jonathanpeppers as a code owner June 17, 2026 09:57

Copilot AI review requested due to automatic review settings June 17, 2026 09:57

simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Jun 17, 2026

Copilot started reviewing on behalf of simonrozsival June 17, 2026 09:58 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread .github/skills/ci-status/SKILL.md Outdated

Comment thread .github/skills/ci-status/references/azdo-queries.md Outdated

Comment thread .github/skills/ci-status/scripts/ci_failures.py Outdated

Comment thread .github/skills/ci-status/SKILL.md Outdated

jonathanpeppers approved these changes Jun 17, 2026

View reviewed changes

jonathanpeppers reviewed Jun 17, 2026

View reviewed changes

simonrozsival removed the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Jun 26, 2026

simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Jun 27, 2026

jonathanpeppers approved these changes Jun 29, 2026

View reviewed changes

simonrozsival merged commit 504b78e into dotnet:main Jun 29, 2026
38 of 40 checks passed

simonrozsival mentioned this pull request Jun 29, 2026

Fast Deployment XA0137 (run-as 'couldn't stat') races install on the PRIMARY user (user 0) ~daily in CI #11808

Open

Copilot AI mentioned this pull request Jun 29, 2026

[FastDeploy] Retry first run-as query to absorb post-install data-dir race (XA0137) #11809

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[skills] Update CI status skill for public pipeline#11670

[skills] Update CI status skill for public pipeline#11670
simonrozsival merged 9 commits into
dotnet:mainfrom
simonrozsival:simonrozsival/update-ci-skills

simonrozsival commented Jun 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonathanpeppers Jun 17, 2026

Uh oh!

simonrozsival Jun 22, 2026

Uh oh!

simonrozsival commented Jun 26, 2026

Uh oh!

simonrozsival commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

simonrozsival commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonathanpeppers Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

simonrozsival Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

simonrozsival commented Jun 26, 2026

Uh oh!

simonrozsival commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simonrozsival commented Jun 16, 2026 •

edited

Loading