Skip to content

FEAT: Support for Apache Arrow in Bulk Copy API #665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
gargsaumya wants to merge 6 commits into
base: main
Choose a base branch
from
+1,563 −93
Draft

FEAT: Support for Apache Arrow in Bulk Copy API #665

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5919288
GH-570: Add Cursor.bulkcopy_arrow for native Arrow bulk copy
gargsaumya Jul 8, 2026
886a30c
Merge remote-tracking branch 'origin/main' into saumya/gh-570
gargsaumya Jul 14, 2026
fd4d490
fix: make bulkcopy_arrow batch_size/timeout positional-or-keyword (de…
gargsaumya Jul 14, 2026
ebf5a5c
test: rename mislabeled Arrow column-mapping test to reflect position…
gargsaumya Jul 17, 2026
a0bf20f
fix(bulkcopy): correct Arrow-steering message (tuples/Row, add __arro…
gargsaumya Jul 17, 2026
d20c1bb
bench: extensive bulkcopy vs bulkcopy_arrow benchmark (profiles, stat…
gargsaumya Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Added
- New feature: Support for macOS and Linux.
- Documentation: Added API documentation in the Wiki.
- **GH-570:** New `Cursor.bulkcopy_arrow(table_name, source)` method for
high-performance bulk loading directly from Apache Arrow data. Accepts a
`pyarrow.Table`, `RecordBatch`, or `RecordBatchReader`, any object exposing
the Arrow C Data Interface (`__arrow_c_stream__` / `__arrow_c_array__` — e.g.
polars, pandas 2.2+, DuckDB, ADBC results), or an iterable of record batches.
Data is streamed to the server through the Arrow C Data Interface without
materializing intermediate Python row objects, and the GIL is released for
the duration of the network transfer. When the source data already originates
as Arrow, this avoids the Arrow→tuple conversion the classic `bulkcopy()`
path requires (measured ~1.4x–2.7x faster end-to-end for such sources).
`bulkcopy()` now raises `TypeError` steering Arrow inputs to this method.
Requires `mssql-py-core` 0.1.5+.
- Bulk copy now supports `Authentication=ActiveDirectoryServicePrincipal`
via an `entra_id_token_factory` callback registered on the mssql-py-core
connection. The callback is invoked by mssql-tds mid-handshake (FedAuth
Expand Down
Loading
Loading