CHORE: Add standalone mssql-python-odbc package for ODBC driver binaries

[jahnvi480]

Work Item / Issue Reference

AB#46217

---

Summary

Splits the ODBC driver binaries out of mssql-python into a new standalone, pure-data package mssql-python-odbc (import name mssql_python_odbc, version 18.6.0). This is the code / packaging half of the split; the ADO release pipelines are in companion PR #664.

What is included

• mssql_python_odbc/__init__.py - pure-Python package (no native extension) that ships the ODBC driver libs/ tree and exposes get_driver_path() and get_libs_dir() with the same per-platform layout the C++ loader expects.
• setup_odbc.py - builds the mssql-python-odbc wheels. Each wheel is platform-specific but Python-agnostic (py3-none-<platform>), so we ship 7 wheels total (Windows amd64/arm64, macOS universal2, manylinux_2_28 x86_64/aarch64, musllinux_1_2 x86_64/aarch64) instead of a per-Python matrix. On Windows the wheel intentionally ships the VC++ runtime (vcredist/msvcp140.dll + license) alongside the driver binaries - see "VC++ runtime" below.
• mssql_python/pybind/ddbc_bindings.cpp - LoadDriverOrThrowException() now resolves the driver / libs base directory via a new GetOdbcLibsBaseDir() helper, which imports mssql_python_odbc when present and falls back to the bundled mssql_python libs when it is not. The fallback is GIL-safe and Alpine/musl-safe.
• tests/test_024_odbc_package_split.py - verifies Python vs C++ driver-path parity.
• .gitignore - ignores local mssql_python_odbc/libs/ build copies and egg-info.

VC++ runtime (vcredist) on Windows

The Windows wheel ships libs/windows/<arch>/vcredist/msvcp140.dll (with its license). This is a deliberate, documented choice - not "parity" with the main wheel's file layout, and not a reversal of a licensing/size decision:

• The main mssql-python wheel also ships the VC++ runtime, just relocated: build.bat copies msvcp140.dll out of vcredist/ to the package root next to the compiled ddbc_bindings extension, and setup.py then excludes the now-duplicate vcredist/ folder. That exclusion is de-duplication, not a removal of the runtime.
• This package has no compiled extension, so there is no package-root .pyd to place the runtime beside. Keeping msvcp140.dll in its original vcredist/ folder keeps the driver binaries and the runtime they were built against together, and keeps the wheel self-contained.
• No Phase-2 runtime skew: msodbcsql18.dll's dependency on msvcp140.dll is satisfied in-process by the mssql-python extension (built /MD, which loads msvcp140.dll from beside the .pyd), so the external-package and bundled-libs paths behave identically. This copy is completeness + attribution, which is why GetOdbcLibsBaseDir()'s completeness check verifies the driver + mssql-auth.dll but not vcredist.

Phasing (no breaking change in this PR)

• Phase 2 (this PR): mssql-python prefers the external package but keeps bundling libs/ as a fallback, so existing installs keep working.
• Phase 3 (future, v2.0.0): remove the bundled libs/ and depend on mssql-python-odbc exclusively.

Testing

• test_024 parity suite: 7 passed, 1 skipped.
• Built mssql_python_odbc-18.6.0-py3-none-win_amd64.whl locally: correct wheel tag, Root-Is-Purelib: false, driver DLLs present, vcredist (msvcp140.dll + license) present, twine check passes.

Notes

Companion PR (release pipelines): #664. Infra registration (ESRP onboarding of mssql-python-odbc, ADO build-definition id) is tracked there.

[Copilot]

Pull request overview

Splits the bundled ODBC driver binaries out of mssql-python into a new standalone data-only package (mssql-python-odbc / mssql_python_odbc), updates the native loader to prefer the external package with a fallback, and adds parity tests to ensure Python and C++ resolve identical driver paths.

Changes:

• Add new mssql_python_odbc package with get_driver_path() / get_libs_dir() and platform/arch/distro detection mirroring the C++ resolver.
• Add setup_odbc.py to build platform-specific, Python-agnostic wheels for the ODBC-binaries package, including a local “sync libs” convenience.
• Update ddbc_bindings.cpp to resolve the ODBC libs base dir via mssql_python_odbc when installed, with fallback to bundled mssql_python libs; add tests validating Python/C++ path parity.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_024_odbc_package_split.py	Adds test coverage validating the new package’s API and Python/C++ driver-path parity.
setup_odbc.py	Introduces a dedicated wheel build script for the new ODBC-binaries package and a transition-time libs sync.
mssql_python/pybind/ddbc_bindings.cpp	Switches native driver/libs base-dir resolution to prefer mssql_python_odbc, falling back to bundled libs.
mssql_python_odbc/__init__.py	Implements the new Python API for locating driver binaries within the standalone package.
.gitignore	Ignores transition-time generated/copy artifacts for the new package (libs/, egg-info).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 51%

🎯 Overall Coverage 80%

📈 Total Lines Covered: 6759 out of 8369 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/ddbc_bindings.cpp (51.4%): Missing lines 1136-1138,1142,1146-1148,1151-1153,1175-1176,1182,1195-1198,1331

Summary

• Total: 37 lines
• Missing: 18 lines
• Coverage: 51%

mssql_python/pybind/ddbc_bindings.cpp

Lines 1132-1157

  1132 // its directory as the base that `GetDriverPathCpp` (and the Windows
  1133 // `mssql-auth.dll` lookup) append `libs` to.
  1134 //
  1135 // During the Phase-2 transition we fall back to the bundled `mssql_python`
! 1136 // directory when the external package is not installed, so a wheel that still
! 1137 // bundles `libs/` keeps working. Importing `mssql_python_odbc` here is
! 1138 // Alpine/musl-safe precisely because it is a separate pure package: it cannot
  1139 // trigger the partially-initialized-module circular import that motivated
  1140 // resolving these paths in C++ in the first place.
  1141 //
! 1142 // (`GetDriverPathCpp` is defined further below; forward-declared here so we can
  1143 // verify the external package actually ships this platform's driver binary.)
  1144 std::string GetDriverPathCpp(const std::string& moduleDir);
  1145 
! 1146 std::string GetOdbcLibsBaseDir() {
! 1147     namespace fs = std::filesystem;
! 1148     // This function calls into the Python C-API (py::module::import, attribute
  1149     // access, casts), so it must run with the GIL held. It is a no-op when the
  1150     // GIL is already held — which it is at the sole current call site, during
! 1151     // module initialization — but acquiring it here self-documents the C-API
! 1152     // dependency and keeps a future GIL-released caller from turning this into a
! 1153     // hard crash.
  1154     py::gil_scoped_acquire gil;
  1155     try {
  1156         py::object module = py::module::import("mssql_python_odbc");
  1157         py::object module_path = module.attr("__file__");

---

Lines 1171-1180

  1171         // unconditionally. Verifying both here keeps this resolver's notion of a
  1172         // usable base dir consistent with what the loader below actually needs,
  1173         // so we never select a directory that would later make the loader throw
  1174         // (e.g. a dir that has msodbcsql18.dll but is missing mssql-auth.dll).
! 1175         std::error_code ec;
! 1176         fs::path externalDriver(GetDriverPathCpp(parentDir.string()));
  1177         bool externalComplete = fs::exists(externalDriver, ec);
  1178 #ifdef _WIN32
  1179         if (externalComplete) {
  1180             fs::path externalAuthDll = externalDriver.parent_path() / "mssql-auth.dll";

---

Lines 1178-1186

  1178 #ifdef _WIN32
  1179         if (externalComplete) {
  1180             fs::path externalAuthDll = externalDriver.parent_path() / "mssql-auth.dll";
  1181             externalComplete = fs::exists(externalAuthDll, ec);
! 1182         }
  1183 #endif
  1184         if (externalComplete) {
  1185             LOG("GetOdbcLibsBaseDir: Using external mssql_python_odbc package - directory='%s'",
  1186                 parentDir.string().c_str());

---

Lines 1191-1202

  1191             parentDir.string().c_str());
  1192         return GetModuleDirectory();
  1193     } catch (const py::error_already_set& e) {
  1194         if (e.matches(PyExc_ModuleNotFoundError)) {
! 1195             // Expected in Phase 2 when the standalone package is not installed.
! 1196             // pybind11 has already fetched and cleared the CPython error
! 1197             // indicator, so re-importing `mssql_python` below is safe.
! 1198             LOG("GetOdbcLibsBaseDir: mssql_python_odbc not installed (%s); "
  1199                 "falling back to bundled libs in mssql_python",
  1200                 e.what());
  1201             return GetModuleDirectory();
  1202         }

---

Lines 1327-1335

  1327     platform = "windows";
  1328     // Normalize x86_64 to x64 for Windows naming
  1329     if (arch == "x86_64")
  1330         arch = "x64";
! 1331     fs::path driverPath =
  1332         basePath / "libs" / platform / arch / ("msodbcsql" MSODBCSQL_VERSION_MAJOR ".dll");
  1333     return driverPath.string();
  1334 
  1335 #else

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.1%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.connection.py: 83.6%
mssql_python.logging.py: 85.5%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[bewithgaurav]

some duplicacies are there & need to be resolved, comments added inline
requesting changes