Speed up Code Coverage Jobs #3907

paulmedynski · 2026-01-21T15:58:00Z

Description

The code coverage pipeline job is inefficient, and can be greatly streamlined. We don't need to publish separate uploads to CodeCov for MDS .NET, .NET Framework, and AKV.

Testing

PR runs will validate the new consolidation steps.
I will manually verify the Azure DevOps test results and coverage, and CodeCov reports.

Copilot

Pull request overview

This pull request optimizes CI/CD pipeline execution by separating unit and integration tests to reduce redundant test execution. The PR introduces a testType parameter that controls which test suites run in different pipeline contexts.

Changes:

Added testType parameter ('All', 'Unit', or 'Integration') to pipeline templates
Modified PR pipelines to run only relevant test suites (Unit tests in Project builds, Integration tests in Package builds)
Updated test execution conditionals to respect the new testType parameter

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
eng/pipelines/sqlclient-pr-project-ref-pipeline.yml	Sets testType to 'Unit' for project reference builds, restricting to unit and functional tests
eng/pipelines/sqlclient-pr-package-ref-pipeline.yml	Sets testType to 'Integration' for package reference builds, restricting to manual tests
eng/pipelines/dotnet-sqlclient-ci-core.yml	Defines testType parameter with default 'All' for CI pipelines to continue running all tests
eng/pipelines/common/templates/steps/run-all-tests-step.yml	Implements conditional test execution based on testType parameter
eng/pipelines/common/templates/stages/ci-run-tests-stage.yml	Propagates testType parameter through stage template
eng/pipelines/common/templates/jobs/ci-run-tests-job.yml	Propagates testType parameter through job template

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml

eng/pipelines/common/templates/steps/run-all-tests-step.yml

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

codecov · 2026-01-21T17:06:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.37%. Comparing base (0ec7af1) to head (6ef1ea2).
⚠️ Report is 6 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (0ec7af1) and HEAD (6ef1ea2). Click for more details.

HEAD has 3 uploads less than BASE

Flag BASE (0ec7af1) HEAD (6ef1ea2)

addons 1 0

netfx 1 0

netcore 1 0

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3907      +/-   ##
==========================================
- Coverage   74.99%   68.37%   -6.63%     
==========================================
  Files         269      291      +22     
  Lines       43246    67384   +24138     
==========================================
+ Hits        32433    46073   +13640     
- Misses      10813    21311   +10498

Flag	Coverage Δ
PR-SqlClient-Project	`68.37% <ø> (?)`
addons	`?`
netcore	`?`
netfx	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

benrr101

Overall, probably a good change. The main issue I have right now is the lumping of functional tests with unit tests. My understanding of them were that they were not unit tests in theory, they were just the simple integration tests that validate they do the most basic stuff. As it turns out a lot of those are unit tests, but not the majority. I was working on the side a bit to move the unit tests from the functional tests and into the unit test project. Ultimately I think we're aligned that we want: unit test, local integration test, remote integration test, stress test, and fuzz test projects. And this change is kinda orthogonal to those goals, without getting in the way of them. So go for it 😆

I think another low-ish hanging fruit is to separate the unit tests and functional tests as separate jobs, like the test sets. That way we don't need to repeat unit and functional tests four times for each platform. They're fast, yes, but not instantaneous, and I imagine we could shave a decent amount of time off a build doing it.

paulmedynski · 2026-01-21T18:05:22Z

You're correct on all counts. I chose to combine the running of Unit and Functional tests because they're mostly unit with some local integration sprinkled in, but mainly because their combined run time is close to the Manual ones. This gives us the most bang-for-the-buck in terms of speeding up the PR runs right now.

We also shouldn't be running Unit and Functional tests in any of the Azure SQL configurations - another future optimization.

eng/pipelines/dotnet-sqlclient-ci-core.yml

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/dotnet-sqlclient-ci-core.yml

priyankatiwari08

Looks good to me. Manual tests in itself takes quite significant amount of time to run. So, this segregation should help us in longer run.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

- Use a temp dir to avoid cluttering up the pooled agents. - Combine all merging/converting into a few files as possible. - Use the modern features of the dotnet tools to avoid parallel jobs and loops. - Combined all MDS reports into a single upload, and same for AKV.

eng/pipelines/dotnet-sqlclient-ci-core.yml

paulmedynski · 2026-01-23T21:01:44Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

    variables:
-      netFxDir: $(Build.SourcesDirectory)\coverageNetFx
-      netCoreDir: $(Build.SourcesDirectory)\coverageNetCore
+      # Use a temp directory that is cleaned up after each job runs.  This helps


I will run the pipeline in debug mode to make sure this temp dir lives on a partition/mount with sufficient space.

Confirmed:

AGENT_TEMPDIRECTORY=/home/vsts/work/_temp Filesystem Size Used Avail Use% Mounted on /dev/root 72G 54G 19G 75% /

The temp dir is on the root partition with plenty of space.

paulmedynski · 2026-01-23T21:03:52Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

-            ${{ else }}:
-              targetPath: $(netCoreDir)
+
+      # Download all of the coverage reports from the test jobs.


I greatly simplified things here. The old tasks went out of their way to separate the MDS .NET and .NET Framework coverage and reports. Now that we've merged the codebases, I can't see why we would care about that. So the new approach is to merge all of the test results together, generate Cobertura reports for MDS and AKV, and then upload them as separate entities. Thoughts?

- Removed unnecessary parameters related to test and coverage publishing.

…'t want coverage of test code.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

tools/targets/ExtractMdsSymbolsFromSnupkg.targets

eng/pipelines/common/templates/steps/run-all-tests-step.yml

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

…since we can't get coverage details from Package reference builds.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

paulmedynski · 2026-01-29T13:39:04Z

eng/pipelines/common/templates/jobs/ci-build-nugets-job.yml

      nuspecPath: 'tools/specs/Microsoft.Data.SqlClient.nuspec'
      OutputDirectory: $(packagePath)
-      generateSymbolsPackage: false
+      generateSymbolsPackage: true


Developers will want symbols packages now that PDBs aren't embedded in the NuGet package.

paulmedynski · 2026-01-29T13:40:55Z

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

        dotnetx86RootPath: $(dotnetx86RootPath)
        operatingSystem: ${{ parameters.operatingSystem }}

-  - ${{ if eq(parameters.publishTestResults, true) }}:


We always want to publish test results.

paulmedynski · 2026-01-29T13:42:09Z

eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml

      --no-build
      -v n
      --filter "category!=failing&category!=flaky"
-      --collect "Code Coverage"


This file is only used in Official pipelines, so no need for coverage reporting.

paulmedynski · 2026-01-29T13:42:21Z

eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml

      --no-build
      -v n
      --filter "category!=failing&category!=flaky"
-      --collect "Code Coverage"


This file is only used in Official pipelines, so no need for coverage reporting.

paulmedynski · 2026-01-29T13:43:03Z

eng/pipelines/common/templates/steps/ci-prebuild-step.yml

      artifact: ${{parameters.artifactName }}
-      patterns: '**/*.nupkg'
-      targetPath: $(Pipeline.Workspace)/${{parameters.artifactName }}
+      patterns: '**/*.*nupkg'


Might as well download the symbols packages too.

paulmedynski · 2026-01-29T13:49:19Z

eng/pipelines/dotnet-sqlclient-ci-core.yml

-      # Include the code coverage job if the build type is Project.
+
+      # Include the code coverage job for Project reference builds.  Coverage
+      # can't be collected properly when we use Package references.


Ok maybe it can, but I went down several rabbit holes and couldn't get coverage from the MDS assembly when it was referenced via package in the tests. We're probably better off adding proper coverage to Unit and Functional tests, and then never using Manual tests to collect coverage.

paulmedynski · 2026-01-29T13:52:39Z

src/Microsoft.Data.SqlClient/add-ons/Directory.Build.props

    <TargetsWindows Condition="'$(OSGroup)'=='Windows_NT'">true</TargetsWindows>
    <TargetsUnix Condition="'$(OSGroup)'=='Unix'">true</TargetsUnix>
    <ReferenceType Condition="'$(ReferenceType)'==''">Project</ReferenceType>
-    <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>


We shouldn't be publishing PDBs with our NuGet packages. We already publish separate symbols packages.

paulmedynski · 2026-01-29T13:53:56Z

src/Microsoft.Data.SqlClient/tests/Directory.Build.props

+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+      <Link>CodeCoverage.runsettings</Link>
+    </None>
+    <None Include="$(MSBuildThisFileDirectory)tools\Microsoft.Data.SqlClient.TestUtilities\xunit.runner.json">


All of the tests need this file, so it made sense to copy it once here rather than multiple times in each project.

paulmedynski · 2026-01-29T13:55:01Z

...t.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/CodeCoverage.runsettings

+            <!-- Include only product assemblies for coverage -->
+            <ModulePaths>
+              <Include>
+                <ModulePath>.*Microsoft\.Data\.SqlClient\.dll$</ModulePath>


This reduces the amount of coverage data generated during test runs to just those assemblies that we care about.

paulmedynski · 2026-01-29T13:55:26Z

...t.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/CodeCoverage.runsettings

Copilot generated this file for me. It seems like a good idea.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

paulmedynski requested a review from a team as a code owner January 21, 2026 15:58

Copilot AI review requested due to automatic review settings January 21, 2026 15:58

paulmedynski added Area\Tests Issues that are targeted to tests or test projects Area\Engineering Use this for issues that are targeted for changes in the 'eng' folder or build systems. labels Jan 21, 2026

Copilot started reviewing on behalf of paulmedynski January 21, 2026 15:58 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml Outdated Show resolved Hide resolved

eng/pipelines/common/templates/steps/run-all-tests-step.yml Outdated Show resolved Hide resolved

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml Outdated Show resolved Hide resolved

benrr101 previously approved these changes Jan 21, 2026

View reviewed changes

paulmedynski dismissed benrr101’s stale review via 546a0eb January 21, 2026 18:20

paulmedynski commented Jan 21, 2026

View reviewed changes

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings January 22, 2026 00:29

Copilot started reviewing on behalf of paulmedynski January 22, 2026 00:29 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

priyankatiwari08 self-assigned this Jan 22, 2026

priyankatiwari08 previously approved these changes Jan 22, 2026

View reviewed changes

paulmedynski dismissed priyankatiwari08’s stale review via e170d4d January 22, 2026 18:35

Copilot AI review requested due to automatic review settings January 22, 2026 18:46

Copilot started reviewing on behalf of paulmedynski January 22, 2026 18:46 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

apoorvdeshmukh previously approved these changes Jan 23, 2026

View reviewed changes

PR pipeline runs will separate which tests they run:

523250b

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot AI review requested due to automatic review settings January 23, 2026 13:52

paulmedynski dismissed apoorvdeshmukh’s stale review via 523250b January 23, 2026 13:52

paulmedynski force-pushed the dev/paul/duplicate-runs branch from 190fcb7 to 523250b Compare January 23, 2026 13:52

Copilot started reviewing on behalf of paulmedynski January 23, 2026 13:53 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

paulmedynski commented Jan 23, 2026

View reviewed changes

- Moved code coverage job to Linux and streamlined it even further.

8b7da18

- Removed unnecessary parameters related to test and coverage publishing.

Added Code Coverage configruation to only include driver code. We don…

358ac5a

…'t want coverage of test code.

Copilot AI review requested due to automatic review settings January 27, 2026 18:45

Copilot started reviewing on behalf of paulmedynski January 27, 2026 18:45 View session

Copilot AI reviewed Jan 27, 2026

View reviewed changes

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml Show resolved Hide resolved

paulmedynski marked this pull request as draft January 28, 2026 11:41

Trying to get code coverage and symbols working for package builds.

f4ae237

paulmedynski changed the title ~~Separate Unit and Integration Test Runs~~ [DRAFT] Separate Unit and Integration Test Runs Jan 28, 2026

Downloading NuGet and Symbols packages for test runs.

1a06f29

Copilot AI review requested due to automatic review settings January 28, 2026 17:00

Copilot started reviewing on behalf of paulmedynski January 28, 2026 17:00 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

tools/targets/ExtractMdsSymbolsFromSnupkg.targets Outdated Show resolved Hide resolved

eng/pipelines/common/templates/steps/run-all-tests-step.yml Outdated Show resolved Hide resolved

paulmedynski added 2 commits January 28, 2026 13:10

Updated pipelines to generate symbols with nuget.

483cf95

Using DotNetCoreCli on Unix instead of MSBuild.

2aa3099

Copilot AI review requested due to automatic review settings January 28, 2026 17:14

Copilot started reviewing on behalf of paulmedynski January 28, 2026 17:15 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

paulmedynski added 2 commits January 28, 2026 15:06

Fixed incorrect MDS NuGet package version in CI runs.

aa195dd

Abandoning running unit and integration tests in separate pipelines, …

66bd89f

…since we can't get coverage details from Package reference builds.

Copilot AI review requested due to automatic review settings January 29, 2026 12:45

Copilot started reviewing on behalf of paulmedynski January 29, 2026 12:45 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

Reverted changes to blame config.

6ef1ea2

paulmedynski commented Jan 29, 2026

View reviewed changes

paulmedynski changed the title ~~[DRAFT] Separate Unit and Integration Test Runs~~ [DRAFT] Speed up Code Coverage Jobs Jan 29, 2026

paulmedynski changed the title ~~[DRAFT] Speed up Code Coverage Jobs~~ Speed up Code Coverage Jobs Jan 29, 2026

paulmedynski marked this pull request as ready for review January 29, 2026 21:04

Copilot AI review requested due to automatic review settings January 29, 2026 21:04

Copilot started reviewing on behalf of paulmedynski January 29, 2026 21:04 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml Show resolved Hide resolved

cheenamalhotra approved these changes Jan 29, 2026

View reviewed changes

Speed up Code Coverage Jobs #3907

Are you sure you want to change the base?

Speed up Code Coverage Jobs #3907

Conversation

paulmedynski commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

benrr101 left a comment

Choose a reason for hiding this comment

Uh oh!

paulmedynski commented Jan 21, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

priyankatiwari08 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

paulmedynski commented Jan 21, 2026 •

edited

Loading

codecov bot commented Jan 21, 2026 •

edited

Loading