Skip to content

[SPARK-56329][PYTHON] Fix all E721 type comparison violations#55150

Closed
zhengruifeng wants to merge 1 commit intoapache:masterfrom
zhengruifeng:fix-isinstance-pandas-types
Closed

[SPARK-56329][PYTHON] Fix all E721 type comparison violations#55150
zhengruifeng wants to merge 1 commit intoapache:masterfrom
zhengruifeng:fix-isinstance-pandas-types

Conversation

@zhengruifeng
Copy link
Copy Markdown
Contributor

@zhengruifeng zhengruifeng commented Apr 2, 2026

What changes were proposed in this pull request?

Fix all E721 (type-comparison) violations across the PySpark codebase and remove E721 from the pyproject.toml ruff ignore list.

Why are the changes needed?

E721 was ignored in pyproject.toml with a TODO comment ("too many for now"). This PR fixes all 78 violations across 24 files so the rule can be enforced going forward.

Changes:

  • type(x) == SomeTypeisinstance(x, SomeType)
  • type(x) != SomeTypenot isinstance(x, SomeType)
  • type(x) == type(y) / type(x) != type(y)type(x) is type(y) / type(x) is not type(y)
  • type(x) in [A, B]isinstance(x, (A, B))
  • variant_type == dictvariant_type is dict (type-identity comparisons)

Does this PR introduce any user-facing change?

No

How was this patch tested?

CI

Was this patch authored or co-authored using generative AI tooling?

Co-authored-by: Claude code (Opus 4.6)

@zhengruifeng
Copy link
Copy Markdown
Contributor Author

cc @gaogaotiantian can you help review? this change was suggested by AI and I remember we discussed this before

@gaogaotiantian
Copy link
Copy Markdown
Contributor

If we want to do this, remove E721 from pyproject.toml and run ruff check - we should fix all occurences at once.

@zhengruifeng zhengruifeng force-pushed the fix-isinstance-pandas-types branch from 143ff27 to d4e925f Compare April 2, 2026 04:03
@zhengruifeng zhengruifeng changed the title [SPARK-56329][PYTHON] Unify the datatype check in python/pyspark/sql/pandas/types.py [SPARK-56329][PYTHON] Fix all E721 type comparison violations and remove E721 from ruff ignore list Apr 2, 2026
@zhengruifeng zhengruifeng changed the title [SPARK-56329][PYTHON] Fix all E721 type comparison violations and remove E721 from ruff ignore list [SPARK-56329][PYTHON] Fix all E721 type comparison violations Apr 2, 2026
@zhengruifeng zhengruifeng force-pushed the fix-isinstance-pandas-types branch from d4e925f to 349d015 Compare April 2, 2026 08:07
@zhengruifeng
Copy link
Copy Markdown
Contributor Author

@gaogaotiantian please take another look

Replace all `type(x) == SomeType` with `isinstance(x, SomeType)` and remove E721 from pyproject.toml ignore list.

Co-authored-by: Isaac
@zhengruifeng zhengruifeng force-pushed the fix-isinstance-pandas-types branch from 349d015 to c3cba34 Compare April 3, 2026 07:27
@zhengruifeng
Copy link
Copy Markdown
Contributor Author

thanks, merged to master

@zhengruifeng zhengruifeng deleted the fix-isinstance-pandas-types branch April 3, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants