Enhance CollectionAssert.AreEqual error messages to include full collections#6611
Enhance CollectionAssert.AreEqual error messages to include full collections#6611Meir017 wants to merge 6 commits intomicrosoft:mainfrom
Conversation
…ections - Update CollectionAssert.AreEqual to include full expected and actual collections in failure messages - Modify FrameworkMessages.resx to add placeholders for full collections - Update localization files (.xlf) with new message format - Add and update unit tests to verify the enhanced error messages Fixes microsoft#334
Evangelink
left a comment
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Thanks for the contribution @Meir017! The idea of including full collections in failure messages is great for debuggability. I have a few concerns that should be addressed before this can be merged:
Critical:
- The full collection stringification happens eagerly on every call — including the happy path where collections match. This is a performance regression for all
AreEqual/AreNotEqualcallers. Please defer to the failure path only.
High:
- No truncation for large collections — risk of OOM / unreadable output
- Hardcoded non-localized string for nullability check — needs a
FrameworkMessagesresource entry - Confusing "Expected"/"Actual" labels in
NumberOfElementsDiff— the labels suggest element counts, but show collection contents
Medium:
fullExpected/fullActualare constant for the entire comparison but duplicated on every stack Tuple entry — they should stay as method-level variables- "Full Expected" / "Full Actual" labels are grammatically awkward — consider "Expected collection" / "Actual collection"
See inline comments for details and suggestions.
| string fullExpectedStr = expected == null ? "null" : string.Join(", ", expected.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture))); | ||
| string fullActualStr = actual == null ? "null" : string.Join(", ", actual.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture))); | ||
|
|
||
| return CompareIEnumerable(expected, actual, comparer, ref reason, fullExpectedStr, fullActualStr); |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Critical — Eager enumeration on the happy path (performance regression)
These two lines fully enumerate both collections and build string representations before the comparison even starts. This means on the happy path (collections are equal — the overwhelming majority of calls), two complete string representations are built and immediately thrown away.
For large collections (thousands of elements), this causes significant allocation pressure and CPU overhead on every single AreEqual / AreNotEqual call, including the ones that pass.
Suggestion: Defer stringification to only happen on the failure path. The ICollection references are already available and can be passed down to CompareIEnumerable. Build the strings only inside the branches that actually set reason. For example:
// Don't do this eagerly — pass the collections and stringify only on failure
return CompareIEnumerable(expected, actual, comparer, ref reason);And then inside CompareIEnumerable, build the strings only when a mismatch is found — or use Lazy<string> to defer.
|
|
||
| return CompareIEnumerable(expected, actual, comparer, ref reason); | ||
| string fullExpectedStr = expected == null ? "null" : string.Join(", ", expected.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture))); | ||
| string fullActualStr = actual == null ? "null" : string.Join(", ", actual.Cast<object>().Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture))); |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
High — No truncation / size limit on collection stringification
If a user compares two collections of 100,000 elements, the error message will contain the entire serialized contents of both collections. This can:
- Produce multi-megabyte error messages
- Cause OOM in constrained test environments
- Make test output completely unreadable
Suggestion: Truncate after a reasonable number of elements (e.g., 10–20) and append "... (N total)". For example:
static string FormatCollection(ICollection collection, int maxElements = 10)
{
var items = collection.Cast<object>().Take(maxElements + 1).ToList();
var formatted = string.Join(", ", items.Take(maxElements).Select(x => Convert.ToString(Assert.ReplaceNulls(x), CultureInfo.InvariantCulture)));
return items.Count > maxElements
? $"{formatted}, ... ({collection.Count} total)"
: formatted;
}| { | ||
| reason = string.Format(CultureInfo.CurrentCulture, "Collections differ in nullability.\nExpected: {0}\nActual: {1}", fullExpected, fullActual); | ||
| return false; | ||
| } |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
High — Hardcoded non-localized error message
This string is hardcoded directly in C# code, bypassing the entire localization system. All other error messages in this method use FrameworkMessages.* resources. This message needs a corresponding entry in FrameworkMessages.resx and all 13 .xlf files.
Fix: Add a new resource entry (e.g., CollectionsDifferInNullability) to FrameworkMessages.resx with value "Collections differ in nullability.\nExpected: {0}\nActual: {1}", add entries in all .xlf files, and reference it as FrameworkMessages.CollectionsDifferInNullability here.
|
|
||
| while (stack.Count > 0) | ||
| { |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Medium — Unnecessary data carried through the stack Tuple
fullExpected and fullActual are constant for the entire comparison — they never change between stack entries. They're already available as method parameters. Carrying them on every stack Tuple entry is redundant overhead and makes the Tuple unwieldy (5 type parameters).
Suggestion: Remove items 4 and 5 from the Tuple and just use the fullExpected / fullActual parameters directly (they're already captured in the method scope):
var stack = new Stack<Tuple<IEnumerator, IEnumerator, int>>();
// ... use fullExpected/fullActual from method parameters directly| <data name="NumberOfElementsDiff" xml:space="preserve"> | ||
| <value>Different number of elements.</value> | ||
| <value>Different number of elements. | ||
| Expected: {0} | ||
| Actual: {1}</value> | ||
| </data> |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
High — Confusing "Expected"/"Actual" labels in NumberOfElementsDiff
The resource now reads:
Different number of elements.
Expected: {0}
Actual: {1}
Here {0} and {1} are the full collection contents (e.g. "1, 2, 3"), not the element counts. A user reading "Different number of elements. Expected: 1, 2, 3" will likely think the expected count was literally "1, 2, 3" rather than understanding they're seeing the collection contents.
Suggestion: Either:
- Include both the counts and the collection contents with distinct labels, or
- Use the same
"Full Expected"/"Full Actual"labeling used inElementsAtIndexDontMatch(though see my comment on that naming too), or - Use more explicit labels like
"Expected collection"/"Actual collection"
| <data name="NumberOfElementsDiff" xml:space="preserve"> | ||
| <value>Different number of elements.</value> | ||
| <value>Different number of elements. | ||
| Expected: {0} | ||
| Actual: {1}</value> | ||
| </data> |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
High — Confusing "Expected"/"Actual" labels in NumberOfElementsDiff
The resource now reads:
Different number of elements.
Expected: {0}
Actual: {1}
Here {0} and {1} are the full collection contents (e.g. "1, 2, 3"), not the element counts. A user reading "Different number of elements. Expected: 1, 2, 3" will likely think the expected count was literally "1, 2, 3" rather than understanding they're seeing the collection contents.
Suggestion: Either:
- Include both the counts and the collection contents with distinct labels, or
- Use the same
"Full Expected"/"Full Actual"labeling used inElementsAtIndexDontMatch(though see my comment on that naming too), or - Use more explicit labels like
"Expected collection"/"Actual collection"
| Full Expected: {3} | ||
| Full Actual: {4}</value> |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Medium — "Full Expected" / "Full Actual" are non-standard labels
The labels "Full Expected" and "Full Actual" are grammatically awkward in English. More natural and clear phrasing would be:
"Expected collection"/"Actual collection", or"Full expected collection"/"Full actual collection"
This is a user-facing string that appears in assertion failure messages. Consistency and clarity matter here.
|
|
||
| public void CollectionAssertAreEqual_DifferentLengths_FailsWithFullCollections() | ||
| { | ||
| Action action = () => CollectionAssert.AreEqual(new[] { 1, 2, 3 }, new[] { 1, 2 }); | ||
| action.Should().Throw<Exception>() | ||
| .And.Message.Should().Be(""" | ||
| CollectionAssert.AreEqual failed. Different number of elements. | ||
| Expected: 1, 2, 3 | ||
| Actual: 1, 2 |
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Low — Inconsistent assertion style in new test
This new test uses .And.Message.Should().Be(...) (exact match) while the existing tests in this file use .WithMessage(...) (wildcard match). The style should be consistent with the surrounding tests.
Additionally, .Throw<Exception>() should probably be .Throw<AssertFailedException>() for specificity, matching what MSTest actually throws.
Also, this test method is missing the [TestMethod] attribute equivalent — but since this project uses TestContainer which discovers all public parameterless void/Task methods, it should be picked up. However, please verify this test is actually being discovered and run.
There was a problem hiding this comment.
[AI generated, reviewed by @Evangelink]
Nit — Whitespace-only changes bloat the diff
The first ~50 lines of this file are trailing-whitespace cleanup in the XML header comments. While individually fine (and consistent with SA1028), these changes add noise to the diff, making the substantive resource changes harder to review. Consider splitting whitespace fixes into a separate commit.
Fixes #334