Skip to content

[python] Support VARIANT type in pypaimon#7635

Open
chenghuichen wants to merge 8 commits intoapache:masterfrom
chenghuichen:python-variant
Open

[python] Support VARIANT type in pypaimon#7635
chenghuichen wants to merge 8 commits intoapache:masterfrom
chenghuichen:python-variant

Conversation

@chenghuichen
Copy link
Copy Markdown
Contributor

@chenghuichen chenghuichen commented Apr 13, 2026

Purpose

Paimon Java already supports the VARIANT type, but pypaimon has no equivalent implementation. This PR adds VARIANT read/write support to pypaimon, enabling compute engines to integrate it. For example, Daft is planning native VARIANT support at the engine level.

Tests

  • Unit tests
    • pypaimon/tests/variant_test.py
  • E2E tests
    • run_java_variant_write_py_read_test
    • run_py_variant_write_java_read_test

@chenghuichen
Copy link
Copy Markdown
Contributor Author

The right long-term solution is to wait for PyArrow's official Variant support (apache/arrow#45937), which would guarantee both correctness and performance from the upstream ecosystem. We should continue tracking that progress.

That said, a Python-based implementation now is a reasonable short-term step: users and compute engines can begin integrating VARIANT support early, and as long as GenericVariant's interface is designed robustly, the underlying implementation can be swapped for PyArrow's later without breaking users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant