feat: implement native native existence join support

[coderfender]

Which issue does this PR close?

Closes #3881 .

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[comphead]

Thanks @coderfender this would also need to be tested by sql tests trying to cover as much edge cases as possible. Mark joins are pretty recently added in DF and might require extra testing

[coderfender]

Absolutely agreed! Thank you for the review @comphead .

[coderfender]

Mixed results with benches

Running benchmark: exists OR predicate (BHJ)
  Running case: Spark
  Stopped after 29 iterations, 2015 ms
  Running case: Comet
  Stopped after 61 iterations, 2011 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                43             70          15         24.4          41.1       1.0X
Comet                                                29             33           3         35.7          28.0       1.5X

Running benchmark: exists OR predicate (SHJ)
  Running case: Spark
  Stopped after 53 iterations, 2029 ms
  Running case: Comet
  Stopped after 44 iterations, 2043 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                35             38           2         29.7          33.7       1.0X
Comet                                                42             46           5         24.8          40.3       0.8X

Running benchmark: exists OR predicate (SMJ)
  Running case: Spark
  Stopped after 54 iterations, 2034 ms
  Running case: Comet
  Stopped after 35 iterations, 2019 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                35             38           2         29.8          33.6       1.0X
Comet                                                54             58           4         19.4          51.6       0.7X

[andygrove]

Thanks @coderfender. The posted benchmarks show that this would be a performance regression compared to Spark? Did performance improve since those were posted?

[comphead]

Apparently sooner or later we would need to use DF mark joins, they can be disabled by default and we can improve them in background

[coderfender]

@andygrove thank you for the comment, Yes the bechmarks are indeed different (with atleast 2x speedup)

Running benchmark: exists OR predicate (BHJ)
  Running case: Spark
  Stopped after 3 iterations, 2120 ms
  Running case: Comet
  Stopped after 7 iterations, 2043 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               703            707           7          1.5         670.1       1.0X
Comet                                               286            292           6          3.7         273.2       2.5X

Running benchmark: exists OR predicate (SHJ)
  Running case: Spark
  Stopped after 3 iterations, 2216 ms
  Running case: Comet
  Stopped after 7 iterations, 2205 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               738            739           1          1.4         703.5       1.0X
Comet                                               309            315           7          3.4         294.4       2.4X

Running benchmark: exists OR predicate (SMJ)
  Running case: Spark
  Stopped after 3 iterations, 2181 ms
  Running case: Comet
  Stopped after 6 iterations, 2017 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               714            727          15          1.5         680.9       1.0X
Comet                                               331            336           4          3.2         315.8       2.2X

[comphead]

nice PR in DF in terms of performance apache/datafusion#23870

[andygrove]

Note on this review: this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.

Mapping Spark's ExistenceJoin onto DataFusion's LeftMark is a neat idea and the change is refreshingly small. Gating it behind a disabled-by-default config while it settles is the right call, and the producedAttributes overrides are the detail that would have been easy to miss.

Before this can be reviewed properly, though, a few things.

The description is empty

Every section of the template is still a placeholder: no rationale, no summary of changes, no testing notes. For a change that introduces a new join type mapping across three join operators, the reviewer needs to know what you verified. In particular I would want the description to answer the semantic question below, because a reader cannot get that from the diff.

Is LeftMark semantically identical to ExistenceJoin?

Spark's ExistenceJoin(exists) emits every left row plus a boolean that is true when at least one right row matches the equi keys and the join condition. Three things I would want confirmed against DataFusion's LeftMark:

• Does LeftMark evaluate the non-equi condition before setting the mark? Spark's ExistenceJoin is most often planned for EXISTS ... OR ... shapes, which is exactly where a non-trivial condition appears. If DataFusion marks on the key match alone and filters afterwards, the results diverge.
• Can the mark column ever be NULL? Spark's exists attribute is nullable = false. If LeftMark can produce a null, downstream expressions will behave differently.
• Where does the mark land in the output schema, and does that match left.output :+ exists?

A short section in the description walking through those, ideally with a DataFusion source link, would make this reviewable.

BroadcastNestedLoopJoinExec is not covered

ExistenceJoin is added to CometHashJoin and CometSortMergeJoinExec, but Spark frequently plans an existence join as a broadcast nested loop join when there are no equi-join keys, which is the common shape for WHERE EXISTS (...) OR x > 1. Is that deliberately out of scope? If so, saying so would help; if not, those queries will silently keep falling back and the feature will look like it does not work.

Benchmark results

CometExistenceJoinBenchmark is added but the description has no numbers. Since the motivation for a native existence join is presumably speed, what does it buy over the fallback?

One CI note

There is a failing check on this PR. Worth looking at before the next round of review.