Skip to content

test(integ-test): stabilize deterministic fixtures across shards - #5730

Open
mengweieric wants to merge 1 commit into
opensearch-project:mainfrom
mengweieric:menwe/multi-shard-deterministic-fixtures
Open

test(integ-test): stabilize deterministic fixtures across shards#5730
mengweieric wants to merge 1 commit into
opensearch-project:mainfrom
mengweieric:menwe/multi-shard-deterministic-fixtures

Conversation

@mengweieric

Copy link
Copy Markdown
Collaborator

Summary

A set of integration tests selected rows through head, limit, shard-local representative choice, or unordered collection output, then asserted exact single-shard results.

This change selects intended fixture rows with unique keys, compares unordered collections as multisets, and uses numeric tolerances only for distributed floating-point or geo-point encoding differences. Schema, cardinality, grouping, source-value, and command-specific assertions remain exact.

No production behavior is modified.

Validation

  • Verified on an external cluster forced to five primary shards
  • Exercised direct, paginating, and independently checked no-pushdown paths as applicable
  • All modified tests passed targeted five-shard validation
  • spotlessCheck, compileTestJava, and git diff --check pass

Select fixture rows by unique keys, compare unordered collections as multisets, and use numeric tolerances where distributed accumulation or encoding is lossy. Keep exact schema, cardinality, and source-value assertions.

Signed-off-by: Eric Wei <menwe@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 No relevant tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review

Incomplete assertion

The test now asserts only 4 of 5 expected rows for David's join result, leaving one row unverified. If max=1 truly collapses David's matches to a single row (total 5), the test should verify all 5 rows or explicitly document why one is omitted. As written, a regression that returns 6 rows (no collapse) would still pass because verifyDataRowsSome only checks presence, not totality.

  verifyDataRowsSome(
      actual2,
      rows("Jake", "England"),
      rows("Jane", "Canada"),
      rows("John", "Canada"),
      rows("Hello", "USA"));
}
Possible Issue

The new matcher jsonEqualsIgnoringOrder treats all arrays as unordered multisets, but graphLookup's connects field (line 499) is a list of airport codes whose order may be semantically meaningful (e.g., representing a traversal sequence). If the order within connects is contractual, comparing it as a multiset silently accepts incorrect orderings. If order is not contractual, this is safe.

private static boolean jsonEqualsIgnoringOrder(Object a, Object b) {
  if (a instanceof JSONArray && b instanceof JSONArray) {
    return jsonArrayEqualsIgnoringNestedOrder((JSONArray) a, (JSONArray) b);
  }
  if (a instanceof JSONObject && b instanceof JSONObject) {
    JSONObject ao = (JSONObject) a;
    JSONObject bo = (JSONObject) b;
    if (ao.keySet().size() != bo.keySet().size()) {
      return false;
    }
    for (String key : ao.keySet()) {
      if (!bo.has(key) || !jsonEqualsIgnoringOrder(ao.get(key), bo.get(key))) {
        return false;
      }
    }
    return true;
  }
  return scalarEquals(a, b);
}

private static boolean jsonArrayEqualsIgnoringNestedOrder(JSONArray actual, JSONArray expected) {
  if (actual.length() != expected.length()) {
    return false;
  }
  List<Object> remaining = new ArrayList<>();
  for (int i = 0; i < expected.length(); i++) {
    remaining.add(expected.get(i));
  }
  for (int i = 0; i < actual.length(); i++) {
    Object actualElement = actual.get(i);
    boolean matched = false;
    for (Iterator<Object> it = remaining.iterator(); it.hasNext(); ) {
      if (jsonEqualsIgnoringOrder(actualElement, it.next())) {
        it.remove();
        matched = true;
        break;
      }
    }
    if (!matched) {
      return false;
    }
  }
  return true;
}

private static boolean scalarEquals(Object a, Object b) {
  boolean aNull = a == null || a == JSONObject.NULL;
  boolean bNull = b == null || b == JSONObject.NULL;
  if (aNull || bNull) {
    return aNull && bNull;
  }
  if (a instanceof Number && b instanceof Number) {
    return ((Number) a).doubleValue() == ((Number) b).doubleValue();
  }
  return a.toString().equals(b.toString());
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Related to improving software testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant