[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862
[dotnet test] Add androidtest project template and dotnet run instrumentation support#10862jonathanpeppers wants to merge 26 commits intomainfrom
androidtest project template and dotnet run instrumentation support#10862Conversation
…trumentation support Context: #10683 Add a new `androidtest` .NET project template that creates an Android test project using MSTest with Microsoft.Testing.Platform. The template includes a TestInstrumentation class that runs tests via `am instrument` and reports results (passed/failed/skipped) back through Android's instrumentation protocol. Add `--instrument` option to Microsoft.Android.Run so `dotnet run` can launch test projects via `am instrument -w` instead of `am start`. Fix an issue where NuGet packages like MSTest add assemblies as both properly-resolved publish assets (with DestinationSubPath metadata) and as None items (without DestinationSubPath) that flow into `_SourceItemsToCopyToPublishDirectory`. When NuGet conflict resolution arbitrarily picks the None-based copy, the assembly loses its DestinationSubPath and TargetPath metadata, causing it to either be missing from the APK or deployed to the wrong directory on device. The fix conditionally removes only items without DestinationSubPath, then re-adds any assemblies that were completely stripped with a cleared TargetPath so that downstream targets (ProcessAssemblies, FastDeploy) can set the correct per-architecture paths. Changes: - New `androidtest` template under src/Microsoft.Android.Templates/ - New GetAndroidInstrumentationName MSBuild task to resolve the instrumentation runner class from AndroidManifest.xml - Updated Microsoft.Android.Sdk.Application.targets to pass `--instrument` when EnableMSTestRunner is true - Added AdbHelper class to Microsoft.Android.Run for shared adb process management - Added `--instrument` CLI option to Microsoft.Android.Run/Program.cs - Fixed _ComputeFilesToPublishForRuntimeIdentifiers to preserve properly-resolved publish assets and recover missing assemblies - Added DotNetNewAndroidTest device integration test TODO: - In a future PR, add support for `dotnet test` end-to-end.
16a61dc to
f121c7c
Compare
Check pe.HasMetadata before calling GetMetadataReader() to avoid InvalidOperationException when processing native DLLs (e.g. from BenchmarkDotNet/TraceEvent NuGet packages) that don't have .NET metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MSTest test platform assemblies fail AOT compilation because the IL linker trims types (DoesNotReturnAttribute, MemberNotNullWhenAttribute) from Microsoft.TestPlatform.CoreUtilities that are still referenced via custom attributes in other test platform DLLs. Use MtouchInterpreter=all to bypass AOT and run all assemblies through the interpreter instead. See: dotnet/android#10862 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tifiers Compare by filename instead of full ItemSpec path when checking if assemblies from _SourceItemsToCopyToPublishDirectory already exist in ResolvedFileToPublish. Native DLLs like KernelTraceControl.dll (from Microsoft.Diagnostics.Tracing.TraceEvent NuGet) can exist under different paths, causing the old Remove-by-path to miss them and re-add duplicates that crash GetPerArchAssemblies with XALNS7004. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…meIdentifiers When _SourceItemsToCopyToPublishDirectory contains multiple items with the same filename from different source paths (e.g., KernelTraceControl.dll), _MissingAssemblyNames ends up with duplicate entries. These duplicates get re-added to ResolvedFileToPublish, causing a Dictionary duplicate key error in GetPerArchAssemblies (XALNS7004). Use RemoveDuplicates task to deduplicate by filename before re-adding, following the same pattern already used for NativeAOT deduplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like microsoft.diagnostics.tracing.traceevent (a
dependency of BenchmarkDotNet) include native Windows DLLs under
lib/native/{arch}/ paths. These were flowing through
_SourceItemsToCopyToPublishDirectory into _MissingAssemblyNames and
being re-added to ResolvedFileToPublish. When the linker tried to
load them with Mono.Cecil, it threw BadImageFormatException
(XALNS7000/XA0009).
Filter out items whose OriginalItemSpec contains a /native/ path
segment, which covers both lib/native/ and runtimes/*/native/ NuGet
layout conventions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet packages like Microsoft.Testing.Extensions.CodeCoverage ship native .so files under runtimes/linux-x64/native/. The .NET SDK stamps RuntimeIdentifier=android-arm64 metadata on all ResolvedFileToPublish items during inner builds, causing ProcessNativeLibraries to treat these non-Android libraries as valid Android native libs. This leads to spurious XA0141 warnings about 16 KB page alignment. Use %(PathInPackage) metadata to detect the actual source RID and exclude .so files from non-Android runtimes (linux-x64, linux-musl-x64, osx-x64, win-x64, etc.) while preserving android-* and linux-bionic-* RIDs which are valid Android targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…native Windows DLLs (#10938) Context: #10862 ## Summary Extracted from PR #10862: Add `PEReader.HasMetadata` checks before calling `GetMetadataReader()` across all code paths that use `PEReader` to prevent exceptions when processing native (non-.NET) PE DLLs. ## Problem When NuGet packages (like MSTest) include native Windows `.dll` files that flow into `ResolvedFileToPublish`, calling `PEReader.GetMetadataReader()` on these files throws an exception because they are not .NET assemblies and have no CLI metadata. ## Fix Added `PEReader.HasMetadata` guards before `GetMetadataReader()` calls across all relevant code paths. In MSBuild tasks with log access, a `LogDebugMessage()` call is included when skipping non-.NET assemblies. Co-authored-by: jonathanpeppers <840039+jonathanpeppers@users.noreply.github.com>
…esToPublishForRuntimeIdentifiers Instead of removing all _SourceItemsToCopyToPublishDirectory items and then recovering missing assemblies through temp item groups and dedup, just set DestinationSubPath on .dll/.pdb items that lack it before the Remove. Items with DestinationSubPath survive; everything else is stripped as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestFramework) add assemblies as @(Reference) items with CopyToOutputDirectory metadata via their .targets files. This causes the assemblies to flow into @(_SourceItemsToCopyToPublishDirectory) in addition to the normal reference resolution path, creating duplicate entries in @(ResolvedFileToPublish) that crash downstream tasks like GetPerArchAssemblies with 'duplicate key' errors. Add _PatchNuGetReferenceMetadata target that strips CopyToOutputDirectory from all @(Reference) items during the inner per-RID publish build, so they only flow through normal reference resolution. This allows reverting _ComputeFilesToPublishForRuntimeIdentifiers back to the simple Remove of @(_SourceItemsToCopyToPublishDirectory). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CancelledTestNodeStateProperty is obsolete (CS0618). Test frameworks should throw OperationCanceledException using the cancellation token passed by Microsoft.Testing.Platform instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Some NuGet packages (e.g. MSTest.TestAdapter) add assemblies as both
@(Reference) items (via buildTransitive .targets) and @(None) items.
The @(None) copy flows into @(_SourceItemsToCopyToPublishDirectory)
and creates a duplicate in @(ResolvedFileToPublish). NuGet conflict
resolution picks arbitrarily between them ('Could not determine winner
due to equal file and assembly versions'). When the @(None)-based copy
wins, _ComputeFilesToPublishForRuntimeIdentifiers removes it via
Remove=@(_SourceItemsToCopyToPublishDirectory), and the assembly is
completely missing from the APK.
Fix by removing @(None) items that match @(Reference) HintPaths in
_PatchNuGetReferenceMetadata, so they never enter the publish pipeline
as duplicates.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This reverts commit 27796e3.
MSTest.TestAdapter adds MSTestAdapter.PlatformServices.dll and Microsoft.TestPlatform.AdapterUtilities.dll only as @(None) items with CopyToOutputDirectory (not as @(Reference)) when EnableMSTestRunner=true. These flow exclusively through @(_SourceItemsToCopyToPublishDirectory) and get blanket-removed from @(ResolvedFileToPublish) by _ComputeFilesToPublishForRuntimeIdentifiers, causing them to be missing from the APK at runtime. Fix by promoting @(None) assembly items with CopyToOutputDirectory to @(Reference) in _PatchNuGetReferenceMetadata, then stripping CopyToOutputDirectory from all References so they flow through normal reference resolution instead of _SourceItemsToCopyToPublishDirectory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Newest The previous condition matched any non-empty CopyToOutputDirectory value, which would incorrectly promote items with CopyToOutputDirectory=Never. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Unqualified %(CopyToOutputDirectory) requires all @(None) items to define the metadata. Items like AndroidManifest.xml don't have it, causing MSB4096. Qualify with %(None.CopyToOutputDirectory) so MSBuild treats missing metadata as empty instead of erroring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion NuGet packages like microsoft.diagnostics.tracing.traceevent add native Windows DLLs (KernelTraceControl.dll, msdia140.dll, etc.) as @(None) items with CopyToOutputDirectory=PreserveNewest and %(Link) containing a platform-specific subdirectory (e.g. amd64\KernelTraceControl.dll). The _PatchNuGetReferenceMetadata target was promoting ALL @(None) .dll items to @(Reference), causing ResolveAssemblyReferences to emit MSB3246 warnings for these native DLLs (PE image does not have metadata). Filter by checking if %(Link) equals its own filename — items with a subdirectory in %(Link) are platform-specific and cannot be used on Android, which packs assemblies flat into a single native library. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds first-class support for Android device test projects driven by MSTest + Microsoft.Testing.Platform by introducing a new androidtest template and enabling dotnet run to launch via Android instrumentation (am instrument) instead of launching an activity.
Changes:
- Adds an
androidtestproject template that includes anInstrumentationentry point which runs tests and reports results via instrumentation bundles. - Extends
Microsoft.Android.Runwith a new--instrumentmode foram instrument -wexecution. - Updates MSBuild targets to locate instrumentation runners from the manifest and adjusts inner-build publish/assembly resolution behavior for certain NuGet package asset patterns.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/MSBuildDeviceIntegration/Tests/InstallAndRunTests.cs | Adds a device integration test validating dotnet new androidtest + dotnet run instrumentation output. |
| src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs | New MSBuild task to extract the instrumentation runner from the manifest. |
| src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.AssemblyResolution.targets | Adds a target that mutates @(None)/@(Reference) metadata to influence publish asset selection. |
| src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.Application.targets | Wires up the new task and passes --instrument to Microsoft.Android.Run when applicable. |
| src/Microsoft.Android.Templates/androidtest/* | New androidtest template files (csproj, manifest, tests, instrumentation runner, template config). |
| src/Microsoft.Android.Run/Program.cs | Adds --instrument option and async instrumentation execution path. |
| src/Microsoft.Android.Run/AdbHelper.cs | Introduces shared adb process start/run helpers. |
|
|
||
| var locker = new Lock (); | ||
| var output = new StringBuilder (); | ||
| var processExited = new ManualResetEventSlim (false); |
There was a problem hiding this comment.
processExited is created but never used. This is dead code and can introduce warnings; please remove it or wire it up (e.g., via process.Exited) if you intended to signal when async output collection is complete.
| var processExited = new ManualResetEventSlim (false); |
| Log.LogError ("The <instrumentation> element is missing the android:name attribute."); | ||
| return false; | ||
| } | ||
| return !Log.HasLoggedErrors; | ||
| } | ||
| } | ||
|
|
||
| Log.LogError ("No <instrumentation> element found in AndroidManifest.xml."); |
There was a problem hiding this comment.
This task logs hard-coded, non-localized errors via Log.LogError(...) and introduces new error text without an XA/APT code. In this repo, MSBuild task errors are expected to use coded errors and localized strings from Properties.Resources (e.g., Log.LogCodedError("XA####", Properties.Resources.XA####, ...)). Please add appropriate resource entries + error code(s) and use coded logging instead of raw strings.
| Log.LogError ("The <instrumentation> element is missing the android:name attribute."); | |
| return false; | |
| } | |
| return !Log.HasLoggedErrors; | |
| } | |
| } | |
| Log.LogError ("No <instrumentation> element found in AndroidManifest.xml."); | |
| Log.LogCodedError ("XA1040", Properties.Resources.XA1040); | |
| return false; | |
| } | |
| return !Log.HasLoggedErrors; | |
| } | |
| } | |
| Log.LogCodedError ("XA1041", Properties.Resources.XA1041); |
| public override bool RunTask () | ||
| { | ||
| using var reader = XmlReader.Create (ManifestFile); | ||
| while (reader.Read ()) { | ||
| if (reader.NodeType == XmlNodeType.Element && reader.LocalName == "instrumentation") { | ||
| InstrumentationName = reader.GetAttribute ("name", ManifestDocument.AndroidXmlNamespace.ToString ()); | ||
| if (InstrumentationName.IsNullOrEmpty ()) { |
There was a problem hiding this comment.
XmlReader.Create(ManifestFile) will throw if the file is missing/invalid XML, which will surface as an unhandled task exception instead of a clean MSBuild error. Consider following the pattern used by GetAndroidActivityName (load via AndroidAppManifest.Load(...)) or explicitly validate ManifestFile exists and catch/convert XML exceptions into a coded MSBuild error.
| <_NoneAssembliesToPromote Include="@(None)" Condition=" '%(None.Extension)' == '.dll' and ('%(None.CopyToOutputDirectory)' == 'Always' or '%(None.CopyToOutputDirectory)' == 'PreserveNewest') and $([System.IO.Path]::GetFileName('%(None.Link)').Equals('%(None.Link)')) " /> | ||
| <None Remove="@(_NoneAssembliesToPromote)" /> | ||
| <Reference Include="@(_NoneAssembliesToPromote)" /> | ||
| <!-- Strip CopyToOutputDirectory from all @(Reference) items (including newly promoted | ||
| ones) so they only flow through reference resolution, not _SourceItemsToCopyToPublishDirectory. | ||
| See: https://github.com/microsoft/testfx/blob/35cc3cd6/src/TestFramework/TestFramework.Extensions/buildTransitive/net8.0AndLater/MSTest.TestFramework.targets#L15-L21 --> | ||
| <Reference Update="@(Reference)" CopyToOutputDirectory="" /> |
There was a problem hiding this comment.
_NoneAssembliesToPromote will match any @(None) item ending in .dll with CopyToOutputDirectory set (including user-authored content/native DLLs with no %(Link)), then promotes it to @(Reference). This can cause ResolveReferences failures (e.g., BadImageFormat) or unintended reference graph changes for non-managed DLLs. Please tighten the criteria to only promote known NuGet package-provided managed assemblies (e.g., items coming from packages/targets, or with explicit metadata) and avoid mutating all @(Reference) items globally if possible.
jonathanpeppers
left a comment
There was a problem hiding this comment.
🤖 AI Review Summary
Verdict:
CI Status: All checks are still pending/in-progress.
Found 4 issues: 1 error, 3 warnings
- ❌ MSBuild tasks:
Log.LogErrorwithout XA error code in GetAndroidInstrumentationName (GetAndroidInstrumentationName.cs:29,36) ⚠️ Nullable: Missing#nullable enablein new AdbHelper.cs (AdbHelper.cs:1)⚠️ Thread safety: Non-atomic counter increments in TestInstrumentation template with parallel execution (TestInstrumentation.cs:93)⚠️ Error handling: Empty catch blocks in RunInstrumentationAsync (Program.cs:235,247)
👍 What's good:
- Clean refactor of sync
RunAdb→ asyncAdbHelper.RunAsyncwith proper async/await throughout - Well-documented
_PatchNuGetReferenceMetadatatarget with clear comments explaining the NuGet assembly resolution fix - Good use of
XmlReader(streaming) and.IsNullOrEmpty()NRT extension in the new MSBuild task - Comprehensive
DotNetNewAndroidTestintegration test covering pass/fail/skip scenarios - Clean mutual exclusion between
--activityand--instrumentwith clear user-facing error messages
Review generated by android-reviewer from review guidelines.
src/Xamarin.Android.Build.Tasks/Tasks/GetAndroidInstrumentationName.cs
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,32 @@ | |||
| using System.Diagnostics; | |||
There was a problem hiding this comment.
🤖 #nullable enable as the very first line, with no preceding blank lines.
Rule: #nullable enable on new files
There was a problem hiding this comment.
We should check if $(Nullable)=enable is set on this entire project or not. If it is, we don't need #nullable enable at the top and we should update instructions/code review rules to skip this comment next time.
src/Microsoft.Android.Templates/androidtest/TestInstrumentation.cs
Outdated
Show resolved
Hide resolved
- GetAndroidInstrumentationName: use Log.LogCodedError with XA1042/XA1043, load manifest via AndroidAppManifest.Load following GetAndroidActivityName pattern, use AndroidAppManifest.AndroidXNamespace - TestInstrumentation: use Interlocked.Increment for thread-safe counters since tests run with method-level parallelization - Program.cs: replace ConfigureAwait(SuppressThrowing) with try/catch for OperationCanceledException, log exceptions in cleanup catch blocks when verbose - InstallAndRunTests: remove unused processExited variable, add process.WaitForExit() after timeout to drain async output events Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Context: #10683
Add a new
androidtest.NET project template that creates an Android test project using MSTest with Microsoft.Testing.Platform. The template includes a TestInstrumentation class that runs tests viaam instrumentand reports results (passed/failed/skipped) back through Android's instrumentation protocol.Add
--instrumentoption to Microsoft.Android.Run sodotnet runcan launch test projects viaam instrument -winstead ofam start.Fix an issue where NuGet packages like MSTest add assemblies as both properly-resolved publish assets (with DestinationSubPath metadata) and as None items (without DestinationSubPath) that flow into
_SourceItemsToCopyToPublishDirectory. When NuGet conflict resolution arbitrarily picks the None-based copy, the assembly loses its DestinationSubPath and TargetPath metadata, causing it to either be missing from the APK or deployed to the wrong directory on device. The fix conditionally removes only items without DestinationSubPath, then re-adds any assemblies that were completely stripped with a cleared TargetPath so that downstream targets (ProcessAssemblies, FastDeploy) can set the correct per-architecture paths.Changes:
androidtesttemplate under src/Microsoft.Android.Templates/--instrumentwhen EnableMSTestRunner is true--instrumentCLI option to Microsoft.Android.Run/Program.csTODO:
dotnet testend-to-end.