Skip to content

[SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3#55146

Closed
ueshin wants to merge 2 commits intoapache:masterfrom
ueshin:issues/SPARK-56327/test_pandas_grouped_map
Closed

[SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3#55146
ueshin wants to merge 2 commits intoapache:masterfrom
ueshin:issues/SPARK-56327/test_pandas_grouped_map

Conversation

@ueshin
Copy link
Copy Markdown
Member

@ueshin ueshin commented Apr 1, 2026

What changes were proposed in this pull request?

This PR updates python/pyspark/sql/tests/pandas/test_pandas_grouped_map.py for pandas 3 behavior in grouped map pandas UDF tests.

The changes are:

  • update the expected boolean inversion in test_supported_types to use ~pdf.bool on pandas 3 while keeping the existing behavior on older pandas versions
  • update several pandas-side expected-value paths to avoid grouping directly by the same in-DataFrame column on pandas 3
  • use copied groupers such as pdf.id.copy() so the grouped pandas input still keeps the grouping columns while preserving the original grouping semantics
  • add comments explaining why the pandas 3 branch uses copied groupers

Why are the changes needed?

In pandas 3, GroupBy.apply drops grouping columns when the grouping key is the same DataFrame column. These tests build expected results by applying the Python function to pandas-grouped data, so the previous expectations no longer match the grouped map input shape seen by Spark in cases that rely on grouping columns remaining present.

The boolean expectation also needs a pandas-3-specific branch because the old scalar-style inversion logic does not match the pandas 3 object being operated on in these tests.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Updated the related tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

@ueshin
Copy link
Copy Markdown
Member Author

ueshin commented Apr 1, 2026

@ueshin
Copy link
Copy Markdown
Member Author

ueshin commented Apr 2, 2026

Thanks! merging to master.

@ueshin ueshin closed this in e9a348e Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants