You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add opt-in numeric Boolean parsing for CSV scans - #17
Add an opt-in Arrow CSV reader mode accepting exact 0 and 1 in Boolean columns alongside true/false.
Keep default parsing unchanged; dispatch once per Boolean column and preserve the text fast path.
Why
Snowflake COPY INTO accepts CSV text 0 and 1 for BOOLEAN columns, while the strict Arrow CSV reader rejected them. This option lets Snowflake-compatible loaders opt in without changing general CSV behavior.
Validation
cargo +1.95.0 test --offline -p arrow-csv --lib: 101 passed.
Local release microbenchmark, 64K single-column rows: strict text ~82.8M rows/s; opt-in text ~83.2M rows/s; opt-in numeric ~94.7M rows/s (one run; within normal benchmark noise).
Narrow live Snowflake COPY INTO probe loaded true,1,0,false as TRUE,TRUE,FALSE,FALSE.
Pre-merge CI audit: the failing jobs are outside this PR diff. cargo fmt reports only parquet/src/encodings/decoding.rs; Linux/Windows tests fail in parquet::bad_data::non_standard_delta_blocks; clippy reports arrow-csv/src/reader/records.rs:672; rustdoc fails in arrow-avro. This PR changes only arrow-csv/src/reader/mod.rs. Focused Arrow CSV tests (101/101), local release benchmark, Snowflake probe, and independent read-only review passed. Merging under the repository's local-validation/bypass policy; the base-branch CI issues remain separate follow-up work.
Post-merge repeat of the same local release microbenchmark (3 runs, single Boolean CSV column, no parallel build): at 64K rows, strict text was 85.4-85.8M rows/s, opt-in text 85.3-85.5M rows/s, and opt-in 0/1 98.1-99.8M rows/s. At 4K rows, strict and opt-in text were both ~73-74M rows/s. The opt-in text difference is within this benchmark's run-to-run noise; the default strict parser remains on its original compile-time branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
0and1in Boolean columns alongsidetrue/false.Why
Snowflake
COPY INTOaccepts CSV text0and1for BOOLEAN columns, while the strict Arrow CSV reader rejected them. This option lets Snowflake-compatible loaders opt in without changing general CSV behavior.Validation
cargo +1.95.0 test --offline -p arrow-csv --lib: 101 passed.cargo +1.95.0 clippy --offline -p arrow-csv --all-targets: passed with existing unrelated warnings.COPY INTOprobe loadedtrue,1,0,falseasTRUE,TRUE,FALSE,FALSE.Independent read-only review: APPROVED (correctness, regressions, performance, API, tests).