Skip to content

Add bounded CSV record filtering to recovery decoder - #21

Merged
osipovartem merged 3 commits into
arrow-csv-snowflake-boolean-aliasesfrom
csv-quote-recovery
Oct 11, 2026
Merged

osipovartem merged 3 commits into
arrow-csv-snowflake-boolean-aliasesfrom
csv-quote-recovery

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • Add an opt-in two-stage record filter to the CSV recovery decoder. Valid rows avoid physical-line bookkeeping; the strict/vectorized CSV path is unchanged.
  • Bound raw recovery record accumulation when a handler opts in, preventing unbounded buffering for unterminated quoted records.
  • Add chunked source-line and record-limit tests plus ignored local throughput benchmarks.

Verification

  • cargo test --locked --offline -p arrow-csv --lib: 131 passed, 8 ignored.
  • Focused 100k-row quoted CSV benchmark: existing recovery handler ~17-19 ms; filtered handler ~18-19 ms. This is the quote-enabled recovery path only.
  • Independent read-only review: approved, no blocking findings.

Dependent DataFusion/Rustice changes will pin this exact commit and add Snowflake COPY diagnostics.

@github-actions github-actions Bot added the arrow label Oct 11, 2026
@osipovartem
osipovartem merged commit ddb3ce8 into arrow-csv-snowflake-boolean-aliases Oct 11, 2026
3 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant