Skip to content

Add equality-delete apply on PyArrow reads - #3805

Open
daviddallakyan2005 wants to merge 2 commits into
apache:mainfrom
daviddallakyan2005:equality-delete-reads
Open

Add equality-delete apply on PyArrow reads#3805
daviddallakyan2005 wants to merge 2 commits into
apache:mainfrom
daviddallakyan2005:equality-delete-reads

Conversation

@daviddallakyan2005

@daviddallakyan2005 daviddallakyan2005 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Closes #3270

Rationale for this change

Index equality delete files and apply them on PyArrow scans so CDC and merge-on-read tables can be read. Equality deletes still cannot be written.

This ports the DeleteFileIndex work from #3285 onto current main, skips bounds prune when an equality field has been dropped, and left anti joins matching equality delete rows after positional deletes. It supersedes #3285, which is open and conflicting. Credit to Alex Stephen for the index. Please close #3285 if this path is preferred.

When a data file has equality deletes, the scan concatenates that file’s record batches, aligns each matching equality-delete table by field id then concats, adds is_null / fill_null (and is_nan for floats) helper columns per key, left anti joins once, then yields the survivors; files with only position deletes still yield each scanner batch.

Are these changes tested?

Yes. Merged main after #3804 so integration-test is not stopped by the deprecated PuffinFile.to_vector() call.

make test
3962 passed, 3 skipped, 1570 deselected

make test-integration PYTEST_ARGS="-v -k test_read_spark_written_puffin_dv"
1 passed, 5534 deselected

Are there any user-facing changes?

Yes. Tables that already store equality deletes can be scanned.

Index equality delete files and apply them on scan so CDC and
merge-on-read tables can be read. Equality deletes still cannot
be written.

Closes apache#3270.

Co-authored-by: Alex Stephen <alexstephen@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Equality Delete support

1 participant