Spark 4.2: `OneRowRelation` in Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a)

[andygrove]

#4142

Describe the bug

On Spark 4.2.0, TPC-DS q77a loses Comet acceleration for a large part of its plan. This showed up while regenerating the TPC-DS plan-stability golden files for the released Spark 4.2.0 (previously the spark-4.2 profile targeted 4.2.0-preview4).

Comparing the approved plan for q77a against the fallback baseline (approved-plans-v2_7/q77a):

	approved-plans-v2_7 (3.4 - 4.1)	approved-plans-v2_7-spark4_2
CometUnion	4	0
bare Union	0	4
non-Comet Exchange / HashAggregate	0	12
Scan OneRowRelation	0	3

Root cause

Spark 4.2 plans a OneRowRelation into the Union branches of this query (it is absent from the plan on 3.4 - 4.1). Comet does not support it:

+-  Scan OneRowRelation [COMET: Scan OneRowRelation is not supported]

That single unsupported leaf cascades:

1. The Union can no longer be converted, so all 4 CometUnion become bare Union.
2. Its Comet children get capped with CometNativeColumnarToRow.
3. The Final / PartialMerge aggregates above the Union then have no Comet partial aggregate producing their intermediate buffers, so the guard in CometBaseAggregate.doConvert (spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:1639) rejects them:

HashAggregate [COMET: Comet aggregate that merges intermediate buffers requires a Comet child
aggregate when the intermediate buffer formats are incompatible with Spark.
Incompatible aggregate function(s): sum]

Expected behavior

q77a should keep the same degree of Comet acceleration on Spark 4.2 that it has on 3.4 - 4.1 (fully native CometUnion + CometHashAggregate).

Additional context

OneRowRelation is already listed in the default value of spark.comet.sparkToColumnar.supportedOperatorList (spark/src/main/scala/org/apache/comet/CometConf.scala:801), so routing it through the Spark-to-Arrow path may be most of the fix. The adding_a_new_spark_version.md contributor guide also calls out "a new OneRowRelation planning path" as a known Spark 4.x behavioral difference.

The regenerated approved-plans-v2_7-spark4_2/q77a golden bakes in the degraded plan so that CI is green for the Spark 4.2.0 bring-up. That golden should be removed (letting q77a fall back to the shared baseline plan) as part of fixing this.