Skip to content

Enhance CollectionAssert.AreEqual error messages to include full collections#6611

Open
Meir017 wants to merge 6 commits intomicrosoft:mainfrom
Meir017:feature/enhance-collection-assert-error-messages
Open

Enhance CollectionAssert.AreEqual error messages to include full collections#6611
Meir017 wants to merge 6 commits intomicrosoft:mainfrom
Meir017:feature/enhance-collection-assert-error-messages

Conversation

@Meir017
Copy link
Copy Markdown

@Meir017 Meir017 commented Sep 27, 2025

  • Update CollectionAssert.AreEqual to include full expected and actual collections in failure messages
  • Modify FrameworkMessages.resx to add placeholders for full collections
  • Update localization files (.xlf) with new message format
  • Add and update unit tests to verify the enhanced error messages

Fixes #334

…ections

- Update CollectionAssert.AreEqual to include full expected and actual collections in failure messages
- Modify FrameworkMessages.resx to add placeholders for full collections
- Update localization files (.xlf) with new message format
- Add and update unit tests to verify the enhanced error messages

Fixes microsoft#334
Copy link
Copy Markdown
Member

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Thanks for the contribution @Meir017! The idea of including full collections in failure messages is great for debuggability. I have a few concerns that should be addressed before this can be merged:

Critical:

  • The full collection stringification happens eagerly on every call — including the happy path where collections match. This is a performance regression for all AreEqual/AreNotEqual callers. Please defer to the failure path only.

High:

  • No truncation for large collections — risk of OOM / unreadable output
  • Hardcoded non-localized string for nullability check — needs a FrameworkMessages resource entry
  • Confusing "Expected"/"Actual" labels in NumberOfElementsDiff — the labels suggest element counts, but show collection contents

Medium:

  • fullExpected/fullActual are constant for the entire comparison but duplicated on every stack Tuple entry — they should stay as method-level variables
  • "Full Expected" / "Full Actual" labels are grammatically awkward — consider "Expected collection" / "Actual collection"

See inline comments for details and suggestions.

Comment on lines +1187 to +1190
string fullExpectedStr = expected == null ? "null" : string.Join(", ", expected.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));
string fullActualStr = actual == null ? "null" : string.Join(", ", actual.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));

return CompareIEnumerable(expected, actual, comparer, ref reason, fullExpectedStr, fullActualStr);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Critical — Eager enumeration on the happy path (performance regression)

These two lines fully enumerate both collections and build string representations before the comparison even starts. This means on the happy path (collections are equal — the overwhelming majority of calls), two complete string representations are built and immediately thrown away.

For large collections (thousands of elements), this causes significant allocation pressure and CPU overhead on every single AreEqual / AreNotEqual call, including the ones that pass.

Suggestion: Defer stringification to only happen on the failure path. The ICollection references are already available and can be passed down to CompareIEnumerable. Build the strings only inside the branches that actually set reason. For example:

// Don't do this eagerly — pass the collections and stringify only on failure
return CompareIEnumerable(expected, actual, comparer, ref reason);

And then inside CompareIEnumerable, build the strings only when a mismatch is found — or use Lazy<string> to defer.


return CompareIEnumerable(expected, actual, comparer, ref reason);
string fullExpectedStr = expected == null ? "null" : string.Join(", ", expected.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));
string fullActualStr = actual == null ? "null" : string.Join(", ", actual.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

High — No truncation / size limit on collection stringification

If a user compares two collections of 100,000 elements, the error message will contain the entire serialized contents of both collections. This can:

  • Produce multi-megabyte error messages
  • Cause OOM in constrained test environments
  • Make test output completely unreadable

Suggestion: Truncate after a reasonable number of elements (e.g., 10–20) and append "... (N total)". For example:

static string FormatCollection(ICollection collection, int maxElements = 10)
{
    var items = collection.Cast<object>().Take(maxElements + 1).ToList();
    var formatted = string.Join(", ", items.Take(maxElements).Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));
    return items.Count > maxElements
        ? $"{formatted}, ... ({collection.Count} total)"
        : formatted;
}

{
reason = string.Format(CultureInfo.CurrentCulture, "Collections differ in nullability.\nExpected: {0}\nActual: {1}", fullExpected, fullActual);
return false;
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

High — Hardcoded non-localized error message

This string is hardcoded directly in C# code, bypassing the entire localization system. All other error messages in this method use FrameworkMessages.* resources. This message needs a corresponding entry in FrameworkMessages.resx and all 13 .xlf files.

Fix: Add a new resource entry (e.g., CollectionsDifferInNullability) to FrameworkMessages.resx with value "Collections differ in nullability.\nExpected: {0}\nActual: {1}", add entries in all .xlf files, and reference it as FrameworkMessages.CollectionsDifferInNullability here.

Comment on lines 1203 to 1205

while (stack.Count > 0)
{
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Medium — Unnecessary data carried through the stack Tuple

fullExpected and fullActual are constant for the entire comparison — they never change between stack entries. They're already available as method parameters. Carrying them on every stack Tuple entry is redundant overhead and makes the Tuple unwieldy (5 type parameters).

Suggestion: Remove items 4 and 5 from the Tuple and just use the fullExpected / fullActual parameters directly (they're already captured in the method scope):

var stack = new Stack<Tuple<IEnumerator, IEnumerator, int>>();
// ... use fullExpected/fullActual from method parameters directly

Comment on lines 237 to 241
<data name="NumberOfElementsDiff" xml:space="preserve">
<value>Different number of elements.</value>
<value>Different number of elements.
Expected: {0}
Actual: {1}</value>
</data>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

High — Confusing "Expected"/"Actual" labels in NumberOfElementsDiff

The resource now reads:

Different number of elements.
Expected: {0}
Actual: {1}

Here {0} and {1} are the full collection contents (e.g. "1, 2, 3"), not the element counts. A user reading "Different number of elements. Expected: 1, 2, 3" will likely think the expected count was literally "1, 2, 3" rather than understanding they're seeing the collection contents.

Suggestion: Either:

  1. Include both the counts and the collection contents with distinct labels, or
  2. Use the same "Full Expected" / "Full Actual" labeling used in ElementsAtIndexDontMatch (though see my comment on that naming too), or
  3. Use more explicit labels like "Expected collection" / "Actual collection"

Comment on lines 237 to 241
<data name="NumberOfElementsDiff" xml:space="preserve">
<value>Different number of elements.</value>
<value>Different number of elements.
Expected: {0}
Actual: {1}</value>
</data>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

High — Confusing "Expected"/"Actual" labels in NumberOfElementsDiff

The resource now reads:

Different number of elements.
Expected: {0}
Actual: {1}

Here {0} and {1} are the full collection contents (e.g. "1, 2, 3"), not the element counts. A user reading "Different number of elements. Expected: 1, 2, 3" will likely think the expected count was literally "1, 2, 3" rather than understanding they're seeing the collection contents.

Suggestion: Either:

  1. Include both the counts and the collection contents with distinct labels, or
  2. Use the same "Full Expected" / "Full Actual" labeling used in ElementsAtIndexDontMatch (though see my comment on that naming too), or
  3. Use more explicit labels like "Expected collection" / "Actual collection"

Comment on lines +196 to +197
Full Expected: {3}
Full Actual: {4}</value>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Medium — "Full Expected" / "Full Actual" are non-standard labels

The labels "Full Expected" and "Full Actual" are grammatically awkward in English. More natural and clear phrasing would be:

  • "Expected collection" / "Actual collection", or
  • "Full expected collection" / "Full actual collection"

This is a user-facing string that appears in assertion failure messages. Consistency and clarity matter here.

Comment on lines +422 to +430

public void CollectionAssertAreEqual_DifferentLengths_FailsWithFullCollections()
{
Action action = () => CollectionAssert.AreEqual(new[] { 1, 2, 3 }, new[] { 1, 2 });
action.Should().Throw<Exception>()
.And.Message.Should().Be("""
CollectionAssert.AreEqual failed. Different number of elements.
Expected: 1, 2, 3
Actual: 1, 2
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Low — Inconsistent assertion style in new test

This new test uses .And.Message.Should().Be(...) (exact match) while the existing tests in this file use .WithMessage(...) (wildcard match). The style should be consistent with the surrounding tests.

Additionally, .Throw<Exception>() should probably be .Throw<AssertFailedException>() for specificity, matching what MSTest actually throws.

Also, this test method is missing the [TestMethod] attribute equivalent — but since this project uses TestContainer which discovers all public parameterless void/Task methods, it should be picked up. However, please verify this test is actually being discovered and run.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI generated, reviewed by @Evangelink]

Nit — Whitespace-only changes bloat the diff

The first ~50 lines of this file are trailing-whitespace cleanup in the XML header comments. While individually fine (and consistent with SA1028), these changes add noise to the diff, making the substantive resource changes harder to review. Consider splitting whitespace fixes into a separate commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide additional information for assert failures

2 participants