Skip to content

[fix](rbo) Guard union equal-set propagation - #67886

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/union-equal-set-propagation
Open

[fix](rbo) Guard union equal-set propagation#67886
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/union-equal-set-propagation

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

UNION ALL could publish output-column equality that does not hold for every row. Downstream rules may then remove a required window ordering key and change RANK() results.

Two forms reproduce the problem:

  1. Regular children project columns in a different order from their internal output. With child filters proving a = b, a union that projects (c, a, b) can incorrectly map that equality to (c, a) and remove a from ORDER BY c, a.
  2. A regular child proves a = b, but constant rows such as (1, 2) and (2, 1) do not. The union can still claim the output columns are equal and remove b from ORDER BY a, b.

Root cause

Logical and physical unions independently mapped child equality sets through child.getOutput() positions. The authoritative union ordinal mapping is regularChildrenOutputs, which can have a different order. The derivation also considered only regular children and ignored every constant row carried by the union.

Reproduction

Create rows (1, 1, 10), (2, 2, 10), and (1, 1, 20), duplicate a filtered (c, a, b) child with UNION ALL, and compute RANK() OVER (ORDER BY c, a). The two rows with (c, a) = (10, 2) must have rank 3.

Separately, union filtered table rows satisfying a = b with constant rows (1, 2) and (2, 1), then compute RANK() OVER (ORDER BY a, b). The four ordered pairs must receive ranks 1, 2, 3, and 4.

Fix

  • Share one equal-set derivation between logical and physical unions.
  • Validate and use each regular child's explicit union-output mapping.
  • Intersect child equality classes by per-ordinal class signatures, avoiding pairwise quadratic candidate generation.
  • Refine candidates against every constant row after SQL comparison coercion and constant folding.
  • Accept only a folded TRUE; NULL, unsupported coercion, non-foldable expressions, and malformed mappings conservatively provide no equality proof.
  • Build each surviving equivalence class with linear star edges.

Tests

  • ./run-fe-ut.sh --run org.apache.doris.nereids.properties.EqualSetTest (13 tests passed)
  • DISABLE_BUILD_UI=ON ./build.sh --fe (passed; Checkstyle reported zero violations)
  • ./run-regression-test.sh --run -f regression-test/suites/nereids_rules_p0/union_equal_set/union_equal_set.groovy (1 suite passed, 0 failed, 0 fatal)

Unit coverage exercises both logical and planner-produced physical unions, reordered mappings, multiple constant rows, one violating row, constant folding, cross-type numeric equality, and NULL. Regression coverage checks both retained window order keys and exact results.

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title Fix union equal-set propagation across all row sources [fix](nereids) Guard union equal-set propagation Sep 11, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 79.43% (112/141) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: UNION ALL could publish output-column equality that did not hold for every row, allowing downstream rules to remove a required window ordering key and change RANK results. Logical and physical unions mapped child equality sets through child output positions instead of the authoritative regular-child ordinal mapping, and they ignored constant rows. Share one derivation that validates the explicit mapping, intersects equality-class signatures across every regular child, and refines candidates against every constant row using SQL coercion and constant folding. Only a folded TRUE proves equality; malformed mappings, NULL, unsupported coercion, and non-foldable expressions fail closed.

### Release note

UNION ALL now propagates column equality only when every regular child and constant row proves it, preventing incorrect downstream ordering simplifications.

### Check List (For Author)

- Test:
    - Unit Test: EqualSetTest (13 tests).
    - Regression test: union_equal_set.
    - Build/checkstyle: DISABLE_BUILD_UI=ON ./build.sh --fe.
- Behavior changed: Yes. Union equality metadata is now withheld unless all row sources prove the equality; valid equalities continue to propagate.
- Does this need documentation: No.
@morrySnow
morrySnow force-pushed the fix/union-equal-set-propagation branch from c3fd287 to 498eaa8 Compare September 11, 2026 21:07
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow morrySnow changed the title [fix](nereids) Guard union equal-set propagation [fix](rbo) Guard union equal-set propagation Sep 12, 2026
@morrySnow
morrySnow marked this pull request as draft September 12, 2026 03:07
hello-stephen pushed a commit that referenced this pull request Sep 12, 2026
…67897)

Since today every `Doris_DorisCloudRegression_VaultP0` run dies in the
`run` step before executing a single test, e.g.
#67883 (TeamCity build 39010) and
#67881 / #67882 / #67885 / #67886 / #67892 / #67893:

```
doris-external--minio Pulling
doris-external--minio Error
Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: start minio docker twice failed
```

MinIO stopped publishing container images in October 2025 (the project
is a source-only distribution now, see minio/minio#21647) and the
`minio/minio` and `minio/mc` repositories have since been removed from
Docker Hub altogether
(`https://hub.docker.com/v2/repositories/minio/minio/` answers 404, same
for `minio/mc`). The few VaultP0 runs that still pass do so only on
agents that have the image cached locally (their logs have no `Pulling`
line). The iceberg, hudi and polaris third-party fixtures,
`test_file_cache_warmup_read_metrics_docker` (which runs a `docker run
minio/minio` itself), the all-in-one `cloud.yml` and the datalake
samples reference the same images and are one cache eviction away from
the same failure.

`quay.io/minio/minio` and `quay.io/minio/mc` still serve every tag we
use -- `RELEASE.2024-11-07T00-52-20Z`, `RELEASE.2025-01-20T14-49-07Z`,
mc `RELEASE.2025-01-17T23-25-50Z`, the two 2022 tags of the samples and
`latest` -- and MinIO keeps pushing hotfix tags there (latest one dated
2026-04). `docker manifest inspect` resolves all of them (amd64 / arm64
/ ppc64le). So every reference gets the `quay.io/` prefix and the tags
stay exactly as they were: same builds, different registry. The CI
agents already pull from quay.io for the OceanBase fixture.

A longer-term option is to mirror these three tags into the project's
own `doristhirdpartydocker` namespace, which already hosts hive /
zookeeper / kafka / trinodb; that needs someone with push access to that
Docker Hub organization and can follow separately.
@morrySnow
morrySnow marked this pull request as ready for review September 12, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants