Skip to content

Preserve missing CSV fields with custom null regex - #10

Merged
osipovartem merged 4 commits into
variant-get-nested-extensionfrom
arrow-csv-truncated-null-presence
Oct 7, 2026
Merged

osipovartem merged 4 commits into
variant-get-nested-extensionfrom
arrow-csv-truncated-null-presence

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 7, 2026 •

Copy link
Copy Markdown

Summary

  • Preserve the number of fields actually present in truncated CSV rows so absent fields remain SQL NULL even when a custom null matcher preserves empty strings. Strict reads allocate no field-count buffer.
  • Add opt-in quoted-empty tracking: "" can remain an empty string while an unquoted empty field maps to NULL. The raw-record scanner and flags are absent from the default path. Schema inference with this mode and an empty null marker requires an explicit schema; otherwise it returns a clear error.
  • Add exact null-value matching with a small-list linear path and a hash set for larger lists. Regex matching remains available for non-literal patterns.

Validation

  • cargo +1.97.0 test --offline -p arrow-csv --lib: 85/85 passed, including chunk boundaries, blank lines, comments, CRLF, custom terminators, truncated rows, escape/quote precedence, and NUL quote.
  • cargo +1.97.0 clippy --offline -p arrow-csv --all-targets -- -D warnings -A clippy::for_kv_map: passed; the suppression is for an unrelated existing warning in arrow-array.
  • cargo +1.97.0 fmt --all -- --check and git diff --check: passed.
  • Local release benchmark, 1M CSV rows, 5 measured iterations: default 47.48 Mrows/s; regex NULL_IF 34.05; exact values 40.70; quoted-empty exact values 27.43. Quoted tracking is opt-in and does not affect the default path.
  • Independent read-only review: approved after addressing the reported parser edge cases.

@osipovartem
osipovartem merged commit 8e80b9c into variant-get-nested-extension Oct 7, 2026
21 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant