Skip to content

[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862

Open
jonathanpeppers wants to merge 26 commits intomainfrom
dev/peppers/androidtest
Open

[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862
jonathanpeppers wants to merge 26 commits intomainfrom
dev/peppers/androidtest

Conversation

@jonathanpeppers
Copy link
Copy Markdown
Member

Context: #10683

Add a new androidtest .NET project template that creates an Android test project using MSTest with Microsoft.Testing.Platform. The template includes a TestInstrumentation class that runs tests via am instrument and reports results (passed/failed/skipped) back through Android's instrumentation protocol.

Add --instrument option to Microsoft.Android.Run so dotnet run can launch test projects via am instrument -w instead of am start.

Fix an issue where NuGet packages like MSTest add assemblies as both properly-resolved publish assets (with DestinationSubPath metadata) and as None items (without DestinationSubPath) that flow into _SourceItemsToCopyToPublishDirectory. When NuGet conflict resolution arbitrarily picks the None-based copy, the assembly loses its DestinationSubPath and TargetPath metadata, causing it to either be missing from the APK or deployed to the wrong directory on device. The fix conditionally removes only items without DestinationSubPath, then re-adds any assemblies that were completely stripped with a cleared TargetPath so that downstream targets (ProcessAssemblies, FastDeploy) can set the correct per-architecture paths.

Changes:

  • New androidtest template under src/Microsoft.Android.Templates/
  • New GetAndroidInstrumentationName MSBuild task to resolve the instrumentation runner class from AndroidManifest.xml
  • Updated Microsoft.Android.Sdk.Application.targets to pass --instrument when EnableMSTestRunner is true
  • Added AdbHelper class to Microsoft.Android.Run for shared adb process management
  • Added --instrument CLI option to Microsoft.Android.Run/Program.cs
  • Fixed _ComputeFilesToPublishForRuntimeIdentifiers to preserve properly-resolved publish assets and recover missing assemblies
  • Added DotNetNewAndroidTest device integration test

TODO:

  • In a future PR, add support for dotnet test end-to-end.

…trumentation support

Context: #10683

Add a new `androidtest` .NET project template that creates an Android
test project using MSTest with Microsoft.Testing.Platform. The template
includes a TestInstrumentation class that runs tests via `am instrument`
and reports results (passed/failed/skipped) back through Android's
instrumentation protocol.

Add `--instrument` option to Microsoft.Android.Run so `dotnet run`
can launch test projects via `am instrument -w` instead of `am start`.

Fix an issue where NuGet packages like MSTest add assemblies as both
properly-resolved publish assets (with DestinationSubPath metadata) and
as None items (without DestinationSubPath) that flow into
`_SourceItemsToCopyToPublishDirectory`. When NuGet conflict resolution
arbitrarily picks the None-based copy, the assembly loses its
DestinationSubPath and TargetPath metadata, causing it to either be
missing from the APK or deployed to the wrong directory on device.
The fix conditionally removes only items without DestinationSubPath,
then re-adds any assemblies that were completely stripped with a cleared
TargetPath so that downstream targets (ProcessAssemblies, FastDeploy)
can set the correct per-architecture paths.

Changes:

- New `androidtest` template under src/Microsoft.Android.Templates/
- New GetAndroidInstrumentationName MSBuild task to resolve the
  instrumentation runner class from AndroidManifest.xml
- Updated Microsoft.Android.Sdk.Application.targets to pass
  `--instrument` when EnableMSTestRunner is true
- Added AdbHelper class to Microsoft.Android.Run for shared adb
  process management
- Added `--instrument` CLI option to Microsoft.Android.Run/Program.cs
- Fixed _ComputeFilesToPublishForRuntimeIdentifiers to preserve
  properly-resolved publish assets and recover missing assemblies
- Added DotNetNewAndroidTest device integration test

TODO:
- In a future PR, add support for `dotnet test` end-to-end.
@jonathanpeppers jonathanpeppers force-pushed the dev/peppers/androidtest branch from 16a61dc to f121c7c Compare March 6, 2026 22:28
jonathanpeppers and others added 3 commits March 6, 2026 16:36
Check pe.HasMetadata before calling GetMetadataReader() to avoid
InvalidOperationException when processing native DLLs (e.g. from
BenchmarkDotNet/TraceEvent NuGet packages) that don't have .NET
metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers added a commit to jonathanpeppers/iOSDotNetTest that referenced this pull request Mar 10, 2026
MSTest test platform assemblies fail AOT compilation because the IL linker
trims types (DoesNotReturnAttribute, MemberNotNullWhenAttribute) from
Microsoft.TestPlatform.CoreUtilities that are still referenced via custom
attributes in other test platform DLLs. Use MtouchInterpreter=all to bypass
AOT and run all assemblies through the interpreter instead.

See: dotnet/android#10862

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers and others added 6 commits March 11, 2026 14:30
…tifiers

Compare by filename instead of full ItemSpec path when checking if
assemblies from _SourceItemsToCopyToPublishDirectory already exist in
ResolvedFileToPublish. Native DLLs like KernelTraceControl.dll (from
Microsoft.Diagnostics.Tracing.TraceEvent NuGet) can exist under
different paths, causing the old Remove-by-path to miss them and
re-add duplicates that crash GetPerArchAssemblies with XALNS7004.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…meIdentifiers

When _SourceItemsToCopyToPublishDirectory contains multiple items with the
same filename from different source paths (e.g., KernelTraceControl.dll),
_MissingAssemblyNames ends up with duplicate entries. These duplicates get
re-added to ResolvedFileToPublish, causing a Dictionary duplicate key error
in GetPerArchAssemblies (XALNS7004).

Use RemoveDuplicates task to deduplicate by filename before re-adding,
following the same pattern already used for NativeAOT deduplication.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like microsoft.diagnostics.tracing.traceevent (a
dependency of BenchmarkDotNet) include native Windows DLLs under
lib/native/{arch}/ paths. These were flowing through
_SourceItemsToCopyToPublishDirectory into _MissingAssemblyNames and
being re-added to ResolvedFileToPublish. When the linker tried to
load them with Mono.Cecil, it threw BadImageFormatException
(XALNS7000/XA0009).

Filter out items whose OriginalItemSpec contains a /native/ path
segment, which covers both lib/native/ and runtimes/*/native/ NuGet
layout conventions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like Microsoft.Testing.Extensions.CodeCoverage ship
native .so files under runtimes/linux-x64/native/. The .NET SDK stamps
RuntimeIdentifier=android-arm64 metadata on all ResolvedFileToPublish
items during inner builds, causing ProcessNativeLibraries to treat
these non-Android libraries as valid Android native libs. This leads
to spurious XA0141 warnings about 16 KB page alignment.

Use %(PathInPackage) metadata to detect the actual source RID and
exclude .so files from non-Android runtimes (linux-x64, linux-musl-x64,
osx-x64, win-x64, etc.) while preserving android-* and linux-bionic-*
RIDs which are valid Android targets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers added a commit that referenced this pull request Mar 16, 2026
…native Windows DLLs (#10938)

Context: #10862

## Summary

Extracted from PR #10862: Add `PEReader.HasMetadata` checks before calling `GetMetadataReader()` across all code paths that use `PEReader` to prevent exceptions when processing native (non-.NET) PE DLLs.

## Problem

When NuGet packages (like MSTest) include native Windows `.dll` files that flow into `ResolvedFileToPublish`, calling `PEReader.GetMetadataReader()` on these files throws an exception because they are not .NET assemblies and have no CLI metadata.

## Fix

Added `PEReader.HasMetadata` guards before `GetMetadataReader()` calls across all relevant code paths. In MSBuild tasks with log access, a `LogDebugMessage()` call is included when skipping non-.NET assemblies.

Co-authored-by: jonathanpeppers <840039+jonathanpeppers@users.noreply.github.com>
jonathanpeppers and others added 13 commits March 16, 2026 08:21
…esToPublishForRuntimeIdentifiers

Instead of removing all _SourceItemsToCopyToPublishDirectory items and
then recovering missing assemblies through temp item groups and dedup,
just set DestinationSubPath on .dll/.pdb items that lack it before the
Remove. Items with DestinationSubPath survive; everything else is
stripped as before.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestFramework) add assemblies as
@(Reference) items with CopyToOutputDirectory metadata via their
.targets files. This causes the assemblies to flow into
@(_SourceItemsToCopyToPublishDirectory) in addition to the normal
reference resolution path, creating duplicate entries in
@(ResolvedFileToPublish) that crash downstream tasks like
GetPerArchAssemblies with 'duplicate key' errors.

Add _PatchNuGetReferenceMetadata target that strips
CopyToOutputDirectory from all @(Reference) items during the inner
per-RID publish build, so they only flow through normal reference
resolution. This allows reverting _ComputeFilesToPublishForRuntimeIdentifiers
back to the simple Remove of @(_SourceItemsToCopyToPublishDirectory).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CancelledTestNodeStateProperty is obsolete (CS0618). Test frameworks
should throw OperationCanceledException using the cancellation token
passed by Microsoft.Testing.Platform instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestAdapter) add assemblies as both
@(Reference) items (via buildTransitive .targets) and @(None) items.
The @(None) copy flows into @(_SourceItemsToCopyToPublishDirectory)
and creates a duplicate in @(ResolvedFileToPublish). NuGet conflict
resolution picks arbitrarily between them ('Could not determine winner
due to equal file and assembly versions'). When the @(None)-based copy
wins, _ComputeFilesToPublishForRuntimeIdentifiers removes it via
Remove=@(_SourceItemsToCopyToPublishDirectory), and the assembly is
completely missing from the APK.

Fix by removing @(None) items that match @(Reference) HintPaths in
_PatchNuGetReferenceMetadata, so they never enter the publish pipeline
as duplicates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MSTest.TestAdapter adds MSTestAdapter.PlatformServices.dll and
Microsoft.TestPlatform.AdapterUtilities.dll only as @(None) items with
CopyToOutputDirectory (not as @(Reference)) when EnableMSTestRunner=true.
These flow exclusively through @(_SourceItemsToCopyToPublishDirectory)
and get blanket-removed from @(ResolvedFileToPublish) by
_ComputeFilesToPublishForRuntimeIdentifiers, causing them to be missing
from the APK at runtime.

Fix by promoting @(None) assembly items with CopyToOutputDirectory to
@(Reference) in _PatchNuGetReferenceMetadata, then stripping
CopyToOutputDirectory from all References so they flow through normal
reference resolution instead of _SourceItemsToCopyToPublishDirectory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Newest

The previous condition matched any non-empty CopyToOutputDirectory value,
which would incorrectly promote items with CopyToOutputDirectory=Never.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Unqualified %(CopyToOutputDirectory) requires all @(None) items to
define the metadata. Items like AndroidManifest.xml don't have it,
causing MSB4096. Qualify with %(None.CopyToOutputDirectory) so MSBuild
treats missing metadata as empty instead of erroring.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion

NuGet packages like microsoft.diagnostics.tracing.traceevent add native
Windows DLLs (KernelTraceControl.dll, msdia140.dll, etc.) as @(None) items
with CopyToOutputDirectory=PreserveNewest and %(Link) containing a
platform-specific subdirectory (e.g. amd64\KernelTraceControl.dll).

The _PatchNuGetReferenceMetadata target was promoting ALL @(None) .dll items
to @(Reference), causing ResolveAssemblyReferences to emit MSB3246 warnings
for these native DLLs (PE image does not have metadata).

Filter by checking if %(Link) equals its own filename — items with a
subdirectory in %(Link) are platform-specific and cannot be used on Android,
which packs assemblies flat into a single native library.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers jonathanpeppers marked this pull request as ready for review April 2, 2026 21:19
Copilot AI review requested due to automatic review settings April 2, 2026 21:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds first-class support for Android device test projects driven by MSTest + Microsoft.Testing.Platform by introducing a new androidtest template and enabling dotnet run to launch via Android instrumentation (am instrument) instead of launching an activity.

Changes:

  • Adds an androidtest project template that includes an Instrumentation entry point which runs tests and reports results via instrumentation bundles.
  • Extends Microsoft.Android.Run with a new --instrument mode for am instrument -w execution.
  • Updates MSBuild targets to locate instrumentation runners from the manifest and adjusts inner-build publish/assembly resolution behavior for certain NuGet package asset patterns.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tests/MSBuildDeviceIntegration/Tests/InstallAndRunTests.cs Adds a device integration test validating dotnet new androidtest + dotnet run instrumentation output.
src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs New MSBuild task to extract the instrumentation runner from the manifest.
src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.AssemblyResolution.targets Adds a target that mutates @(None)/@(Reference) metadata to influence publish asset selection.
src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.Application.targets Wires up the new task and passes --instrument to Microsoft.Android.Run when applicable.
src/Microsoft.Android.Templates/androidtest/* New androidtest template files (csproj, manifest, tests, instrumentation runner, template config).
src/Microsoft.Android.Run/Program.cs Adds --instrument option and async instrumentation execution path.
src/Microsoft.Android.Run/AdbHelper.cs Introduces shared adb process start/run helpers.


var locker = new Lock ();
var output = new StringBuilder ();
var processExited = new ManualResetEventSlim (false);
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

processExited is created but never used. This is dead code and can introduce warnings; please remove it or wire it up (e.g., via process.Exited) if you intended to signal when async output collection is complete.

Suggested change
var processExited = new ManualResetEventSlim (false);

Copilot uses AI. Check for mistakes.
Comment on lines +29 to +36
Log.LogError ("The <instrumentation> element is missing the android:name attribute.");
return false;
}
return !Log.HasLoggedErrors;
}
}

Log.LogError ("No <instrumentation> element found in AndroidManifest.xml.");
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This task logs hard-coded, non-localized errors via Log.LogError(...) and introduces new error text without an XA/APT code. In this repo, MSBuild task errors are expected to use coded errors and localized strings from Properties.Resources (e.g., Log.LogCodedError("XA####", Properties.Resources.XA####, ...)). Please add appropriate resource entries + error code(s) and use coded logging instead of raw strings.

Suggested change
Log.LogError ("The <instrumentation> element is missing the android:name attribute.");
return false;
}
return !Log.HasLoggedErrors;
}
}
Log.LogError ("No <instrumentation> element found in AndroidManifest.xml.");
Log.LogCodedError ("XA1040", Properties.Resources.XA1040);
return false;
}
return !Log.HasLoggedErrors;
}
}
Log.LogCodedError ("XA1041", Properties.Resources.XA1041);

Copilot uses AI. Check for mistakes.
Comment on lines +22 to +28
public override bool RunTask ()
{
using var reader = XmlReader.Create (ManifestFile);
while (reader.Read ()) {
if (reader.NodeType == XmlNodeType.Element && reader.LocalName == "instrumentation") {
InstrumentationName = reader.GetAttribute ("name", ManifestDocument.AndroidXmlNamespace.ToString ());
if (InstrumentationName.IsNullOrEmpty ()) {
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XmlReader.Create(ManifestFile) will throw if the file is missing/invalid XML, which will surface as an unhandled task exception instead of a clean MSBuild error. Consider following the pattern used by GetAndroidActivityName (load via AndroidAppManifest.Load(...)) or explicitly validate ManifestFile exists and catch/convert XML exceptions into a coded MSBuild error.

Copilot uses AI. Check for mistakes.
Comment on lines +56 to +62
<_NoneAssembliesToPromote Include="@(None)" Condition=" '%(None.Extension)' == '.dll' and ('%(None.CopyToOutputDirectory)' == 'Always' or '%(None.CopyToOutputDirectory)' == 'PreserveNewest') and $([System.IO.Path]::GetFileName('%(None.Link)').Equals('%(None.Link)')) " />
<None Remove="@(_NoneAssembliesToPromote)" />
<Reference Include="@(_NoneAssembliesToPromote)" />
<!-- Strip CopyToOutputDirectory from all @(Reference) items (including newly promoted
ones) so they only flow through reference resolution, not _SourceItemsToCopyToPublishDirectory.
See: https://github.com/microsoft/testfx/blob/35cc3cd6/src/TestFramework/TestFramework.Extensions/buildTransitive/net8.0AndLater/MSTest.TestFramework.targets#L15-L21 -->
<Reference Update="@(Reference)" CopyToOutputDirectory="" />
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_NoneAssembliesToPromote will match any @(None) item ending in .dll with CopyToOutputDirectory set (including user-authored content/native DLLs with no %(Link)), then promotes it to @(Reference). This can cause ResolveReferences failures (e.g., BadImageFormat) or unintended reference graph changes for non-managed DLLs. Please tighten the criteria to only promote known NuGet package-provided managed assemblies (e.g., items coming from packages/targets, or with explicit metadata) and avoid mutating all @(Reference) items globally if possible.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

@jonathanpeppers jonathanpeppers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review Summary

Verdict: ⚠️ Needs Changes

CI Status: All checks are still pending/in-progress.

Found 4 issues: 1 error, 3 warnings

  • MSBuild tasks: Log.LogError without XA error code in GetAndroidInstrumentationName (GetAndroidInstrumentationName.cs:29,36)
  • ⚠️ Nullable: Missing #nullable enable in new AdbHelper.cs (AdbHelper.cs:1)
  • ⚠️ Thread safety: Non-atomic counter increments in TestInstrumentation template with parallel execution (TestInstrumentation.cs:93)
  • ⚠️ Error handling: Empty catch blocks in RunInstrumentationAsync (Program.cs:235,247)

👍 What's good:

  • Clean refactor of sync RunAdb → async AdbHelper.RunAsync with proper async/await throughout
  • Well-documented _PatchNuGetReferenceMetadata target with clear comments explaining the NuGet assembly resolution fix
  • Good use of XmlReader (streaming) and .IsNullOrEmpty() NRT extension in the new MSBuild task
  • Comprehensive DotNetNewAndroidTest integration test covering pass/fail/skip scenarios
  • Clean mutual exclusion between --activity and --instrument with clear user-facing error messages

Review generated by android-reviewer from review guidelines.

@@ -0,0 +1,32 @@
using System.Diagnostics;
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 ⚠️ Nullable — New files should have #nullable enable as the very first line, with no preceding blank lines.

Rule: #nullable enable on new files

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check if $(Nullable)=enable is set on this entire project or not. If it is, we don't need #nullable enable at the top and we should update instructions/code review rules to skip this comment next time.

- GetAndroidInstrumentationName: use Log.LogCodedError with XA1042/XA1043,
  load manifest via AndroidAppManifest.Load following GetAndroidActivityName
  pattern, use AndroidAppManifest.AndroidXNamespace
- TestInstrumentation: use Interlocked.Increment for thread-safe counters
  since tests run with method-level parallelization
- Program.cs: replace ConfigureAwait(SuppressThrowing) with try/catch for
  OperationCanceledException, log exceptions in cleanup catch blocks when
  verbose
- InstallAndRunTests: remove unused processExited variable, add
  process.WaitForExit() after timeout to drain async output events

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants