[6.1] Port Flaky Test Separation by benrr101 · Pull Request #4060 · dotnet/SqlClient

benrr101 · 2026-03-17T22:30:57Z

Description

Since 6.1 branch already has the flaky test traits, this PR just updates the PR pipeline to run tests w/o flaky tests, then run only flaky tests, ignoring the results of the latter.

🤖

Codex ported these changes from the branch for #4059

Testing

These changes only apply to PR pipelines. They will be validated here.

Copilot

Pull request overview

This PR updates the shared PR pipeline test template to run additional “flaky-only” Functional and Manual test passes (and ignore failures from those passes) to help separate flaky test failures from main PR signal.

Changes:

Adds “Run Flaky Functional Tests” and “Run Flaky Manual Tests” steps for both Windows (MSBuild@1) and non-Windows (DotNetCoreCLI@2).
Adjusts test-step conditions to succeededOrFailed() so later test steps still execute even when earlier steps fail.
Marks flaky-only steps with continueOnError: true to ignore flaky test failures.

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+    displayName: 'Run Flaky Functional Tests ${{parameters.msbuildArchitecture }}'
+    inputs:
+      solution: build.proj
+      msbuildArchitecture: ${{parameters.msbuildArchitecture }}
+      platform: '${{parameters.platform }}'
+      configuration: '${{parameters.configuration }}'
+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+      ${{ else }}: # x86
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'
+    condition: succeededOrFailed()


eng/pipelines/common/templates/steps/run-all-tests-step.yml

+    displayName: 'Run Flaky Manual Tests ${{parameters.msbuildArchitecture }}'
+    inputs:
+      solution: build.proj
+      msbuildArchitecture: ${{parameters.msbuildArchitecture }}
+      platform: '${{parameters.platform }}'
+      configuration: '${{parameters.configuration }}'
+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:
+        msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+      ${{ else }}: # x86
+        msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'


eng/pipelines/common/templates/steps/run-all-tests-step.yml

        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }}'
      ${{ else }}: # x86
        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }}'


eng/pipelines/common/templates/steps/run-all-tests-step.yml

+  - task: DotNetCoreCLI@2
+    displayName: 'Run Flaky Functional Tests'
+    inputs:
+      command: custom
+      projects: build.proj
+      custom: msbuild
+      arguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:platform=${{parameters.platform }} -p:Configuration=${{parameters.configuration }} -p:Filter="category=flaky"'
+      verbosityRestore: Detailed
+      verbosityPack: Detailed
+    condition: succeededOrFailed()
+    continueOnError: true


eng/pipelines/common/templates/steps/run-all-tests-step.yml

+  - task: DotNetCoreCLI@2
+    displayName: 'Run Flaky Manual Tests'
+    inputs:
+      command: custom
+      projects: build.proj
+      custom: msbuild
+      arguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:platform=${{parameters.platform }} -p:Configuration=${{parameters.configuration }} -p:Filter="category=flaky"'
+      verbosityRestore: Detailed
+      verbosityPack: Detailed
+    condition: succeededOrFailed()
+    continueOnError: true


eng/pipelines/common/templates/steps/run-all-tests-step.yml

Copilot

Pull request overview

This PR aims to port the “flaky test separation” behavior into the 6.1 PR pipeline, so CI runs non-flaky tests normally and then runs flaky tests separately without failing the PR.

Changes:

Updates the shared run-all-tests-step.yml template to add dedicated “Run Flaky * Tests” steps that continue on error.
Adjusts pipeline step conditions to run subsequent test phases even if an earlier phase fails.
Updates ConnectionFailoverTests attributes/traits related to the flaky category.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionFailoverTests.cs	Trait/attribute adjustments for flaky categorization in this unit test class.
eng/pipelines/common/templates/steps/run-all-tests-step.yml	Adds separate flaky test executions for unit/functional/manual tests and tweaks step conditions/behavior.

paulmedynski · 2026-03-19T19:44:04Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+        ${{ else }}: #x86
+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'


This is correct. You need the parts of build.proj from main that support the <Filter> property.

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+        command: custom
+        projects: build.proj
+        custom: msbuild
+        arguments: '-t:RunUnitTests -p:TF=${{ parameters.targetFramework }} -p:TestSet=${{ parameters.testSet }} -p:=TestMicrosoftDataSqlClientVersion=${{ parameters.nugetPackageVersion }} -p:platform=${{ parameters.platform }} -p:Configuration=${{ parameters.configuration }} -p:Filter="category=flaky"'


paulmedynski · 2026-03-19T19:46:31Z

src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionFailoverTests.cs

+        [Trait("Category", "flaky")]
        [Theory]
        [InlineData(40613)]


Agreed - the entire class is flaky, so individual tests don't also need to be marked as flaky.

mdaigle · 2026-03-19T17:52:06Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

        platform: '${{parameters.platform }}'
        configuration: '${{parameters.configuration }}'
        ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:   
          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }}'


Last thing, we'll need to filter out flaky tests for the non-flaky test steps.

This is already done in build.proj via the RunUnitTests target.

What this PR needs is the newer build.proj with the <Filter> property.

Port pipeline changes for flaky tests to 6.1

89c94a2

benrr101 added this to the 6.1.5 milestone Mar 17, 2026

benrr101 requested a review from a team as a code owner March 17, 2026 22:30

Copilot AI review requested due to automatic review settings March 17, 2026 22:30

benrr101 added Area\Tests Issues that are targeted to tests or test projects Area\Engineering Use this for issues that are targeted for changes in the 'eng' folder or build systems. labels Mar 17, 2026

github-project-automation bot added this to SqlClient Board Mar 17, 2026

github-project-automation bot moved this to To triage in SqlClient Board Mar 17, 2026

Copilot started reviewing on behalf of benrr101 March 17, 2026 22:31 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

mdaigle requested changes Mar 18, 2026

View reviewed changes

eng/pipelines/common/templates/steps/run-all-tests-step.yml Show resolved Hide resolved

github-project-automation bot moved this from To triage to In progress in SqlClient Board Mar 18, 2026

Flaky unit tests

ecda665

mdaigle reviewed Mar 18, 2026

View reviewed changes

eng/pipelines/common/templates/steps/run-all-tests-step.yml Show resolved Hide resolved

Run flaky unit tests on windows too

cc60401

Copilot AI review requested due to automatic review settings March 19, 2026 16:44

Copilot started reviewing on behalf of benrr101 March 19, 2026 16:44 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

mdaigle reviewed Mar 19, 2026

View reviewed changes

paulmedynski self-assigned this Mar 19, 2026

paulmedynski requested changes Mar 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[6.1] Port Flaky Test Separation#4060

[6.1] Port Flaky Test Separation#4060
benrr101 wants to merge 3 commits intorelease/6.1from
dev/russellben/flaky/6.1

benrr101 commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

paulmedynski Mar 19, 2026

Uh oh!

paulmedynski Mar 19, 2026

Uh oh!

mdaigle Mar 19, 2026

Uh oh!

paulmedynski Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

benrr101 commented Mar 17, 2026

Description

🤖

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

paulmedynski Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

mdaigle Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants