Skip to content

Join swap in join_selection flips schema-level metadata, failing "Schema mismatch" #23461

Description

@fangchenli

Describe the bug

When join reordering swaps a join's inputs, the swapped plan keeps a different value for a conflicting schema-level metadata key than the original, so OptimizationInvariantChecker rejects it:

Internal error: PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.

swap_inputs rebuilds the join with inputs reversed and wraps it in a reverting ProjectionExec (reorder_output_after_swap), which restores column order and field metadata but not the schema-level metadata. So reversing two inputs that disagree on a key flips the winner, and the pre/post-swap schemas differ. This is on main, independent of merge direction, and shared by cross / hash / NLJ joins.

To Reproduce

// l: 1000 rows, schema metadata {"pandas": "LEFT"}
// r: 2 rows,    schema metadata {"pandas": "RIGHT"}
ctx.sql("SELECT * FROM l CROSS JOIN r").await?.collect().await?;
// -> Internal error: ... 'join_selection' failed. Schema mismatch.

set datafusion.optimizer.join_reordering = false; makes it pass.

Expected behavior

A swap must preserve the output schema, including schema-level metadata.

Additional context

Found while fixing the cross-join initial-plan case in #23434 / PR #23442, which leaves the shared swap path untouched.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions