feat: Add IBM Db2 engine adapter - #6030
Open
hdm-db2-eco-system wants to merge 1 commit into
Open
Conversation
Adds a complete IBM Db2 LUW engine adapter for SQLMesh, including
connection configuration, Docker-based CI, integration tests, and
engine-specific documentation.
## What is included
### Core adapter (sqlmesh/core/engine_adapter/db2.py)
- Db2EngineAdapter implementing all standard SQLMesh engine operations
- SYSCAT-based metadata queries (columns, tables, schemas, indexes, views)
- CTAS with mandatory WITH DATA clause
- MERGE INTO with TARGET/SOURCE alias substitution
- CREATE INDEX with SYSCAT.INDEXES existence check
- CREATE/DROP SCHEMA using SYSCAT.SCHEMATA
- DROP VIEW using SYSCAT.VIEWS existence check
- GRANT/REVOKE using SYSCAT.TABAUTH (not INFORMATION_SCHEMA)
- Truncate implemented as DELETE FROM (Db2 has no TRUNCATE)
- TIMESTAMPTZ stripped to TIMESTAMP (SQL0180N fix)
- normalize_identifiers before quoting (SQL0204N fix)
- catalog_support = SINGLE_CATALOG_ONLY (Db2 databases are isolated)
- SUPPORTED_DROP_CASCADE_OBJECT_KINDS = [] (SQL0104N fix)
- MAX_IDENTIFIER_LENGTH = 128
- set_current_catalog via CONNECT TO
### Connection config (sqlmesh/core/config/connection.py)
- Db2ConnectionConfig with host/port/database/username/password/db2_schema
- SSL options (ssl, ssl_cert, ssl_key, ssl_ca)
- get_catalog() returns .upper() to match Db2 UPPERCASE normalisation
- Excluded from FORBIDDEN_STATE_SYNC_ENGINES with explanation comment
(Db2 rejects table names starting with underscore)
### Framework integration
- sqlmesh/core/engine_adapter/__init__.py: conditional import guarded
by Python >= 3.10 AND find_spec('db2_sqlglot') — prevents import
crash in environments without the db2 extra installed
- sqlmesh/utils/migration.py: db2 added to MAX_TEXT_INDEX_LENGTH
(255) and blob_text_type() returns VARCHAR(32000)
## CI / infrastructure
- .github/workflows/pr.yaml: db2 added to engine-tests-docker matrix
- .github/scripts/install-prerequisites.sh: db2 installs libxml2-dev
- .github/scripts/wait-for-db.sh: db2_ready() readiness probe
- tests/.../docker/compose.db2.yaml: IBM Db2 Community Edition image
- Makefile: db2-test target with junitxml; db2 added to install-dev
- pyproject.toml: db2 optional extra + pytest marker
## Tests
- tests/core/engine_adapter/test_db2.py: 22 unit tests (all passing)
- tests/core/engine_adapter/integration/test_integration_db2.py:
18 Db2-specific integration tests
- tests/core/engine_adapter/integration/__init__.py: SYSCAT comment
queries, role-based grant infrastructure for Db2
- tests/core/engine_adapter/integration/config.yaml: inttest_db2
gateway with DuckDB state_connection
- tests/core/engine_adapter/integration/test_integration.py:
skip blocks with documented reasons for 20 tests; adaptations
for uppercase identifiers, TIMESTAMPTZ, grants
## CI results (stable baseline on Docker Db2 Community Edition)
- 81 passed · 46 skipped · 0 failed
## Known limitations and skipped tests
### SCD Type 2 (4 tests skipped)
Db2 SQL preprocessor treats identifiers starting with '_' as
conditional compilation directives (SQL20521N reason 7). SQLMesh
generates _exists, _key0, _row_number, _t as aliases. Additionally
SCD staging uses CTAS which Db2 does not support natively.
Fix: override _scd_type_2() in Db2EngineAdapter to rename aliases.
### View comments (3 tests skipped)
Db2 has no COMMENT ON VIEW statement (SQL0104N).
### test_sushi (1 test skipped)
CREATE SCHEMA IF NOT EXISTS is not valid Db2 SQL (SQL0104N).
Fix: add create_sql() override to db2-sqlglot-dialect to strip
IF NOT EXISTS from CREATE SCHEMA.
### ctx.create_context() tests (6 tests skipped)
shared.py:346 uses a case-sensitive == comparison between
catalog_name (TESTDB, Db2 uppercase) and _default_catalog
(testdb, duckdb-dialect lowercase). This is a one-line upstream
fix in shared.py that cannot be made in this PR:
catalog_name.upper() != (engine_adapter._default_catalog or '').upper()
## Documentation
- docs/integrations/engines/db2.md: connection options, state
connection guidance, limitations, example config
- docs/integrations/overview.md: Db2 entry added
- docs/guides/connections.md: Db2 link added
- mkdocs.yml: db2.md nav entry added
Signed-off-by: IBM Db2 Eco System <Hdm-dev-persona-db2-eco-system@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a IBM Db2 engine adapter for SQLMesh, including connection
configuration, Docker-based CI, integration tests, and engine-specific documentation.
CI results (stable baseline — Docker Db2 Community Edition)
81 passed · 46 skipped · 0 failed
Tested against:
icr.io/db2_community/db2:latest(Db2 11.5)What works
Model kinds
FULLINCREMENTAL_BY_TIME_RANGEINCREMENTAL_BY_UNIQUE_KEYMERGE INTOconfirmedINCREMENTAL_BY_PARTITIONINCREMENTAL_UNMANAGEDVIEWSEEDSCD_TYPE_2_BY_TIMESCD_TYPE_2_BY_COLUMNCore engine operations
CREATE TABLENOT NULLautomaticallyCTASWITH DATAclauseMERGE INTOTARGET/SOURCEalias substitutionCREATE INDEXSYSCAT.INDEXESfirstCREATE/DROP SCHEMASYSCAT.SCHEMATADROP VIEWSYSCAT.VIEWSGRANT/REVOKESYSCAT.TABAUTH(Db2 has noINFORMATION_SCHEMA.TABLE_PRIVILEGES)TRUNCATEDELETE FROM(Db2 has no native TRUNCATE)COMMENT ON TABLE/COMMENT ON COLUMNSYSCATcatalog viewsWhat is skipped and why
All skips are in
tests/core/engine_adapter/integration/test_integration.pywith documented
pytest.skip()reasons. Search forif ctx.dialect == "db2"in that file to see every skip block.
1. SCD Type 2 — 4 tests skipped
Tests:
test_scd_type_2_by_time,test_scd_type_2_by_time_source_columns,test_scd_type_2_by_column,test_scd_type_2_by_column_source_columnsError:
SQL20521N reason 7— Db2's SQL preprocessor treats identifiersstarting with
_as conditional compilation directives.Root cause: SQLMesh's
_scd_type_2()inbase.pygenerates_exists,_key0,_row_number,_tas column aliases — all underscore-prefixed.The SCD staging table also uses CTAS which Db2 does not support natively.
2. View comments — 3 tests skipped
Tests:
test_create_view,test_create_view_source_columns,test_get_data_objectsError:
SQL0104N— Db2 has noCOMMENT ON VIEWstatement.3.
test_sushi— 1 test skippedError:
SQL0104N—CREATE SCHEMA IF NOT EXISTSis not valid Db2 SQL.The
before_allstatements in the sushi example project emitIF NOT EXISTSunconditionally through the
db2-sqlglot-dialectgenerator.4.
ctx.create_context()tests — 6 tests skippedTests:
test_batch_size_on_incremental_by_unique_key_model,test_state_migrate_from_scratch,test_python_model_column_order,test_unicode_characters,test_grants_plan,test_incremental_by_unique_key_model_when_matchedError:
SQLMeshError: Cannot create a snapshot in a different catalogRoot cause:
shared.py:346performs a case-sensitive==comparisonbetween
catalog_name(uppercased by Db2 normalisation, e.g."TESTDB")and
_default_catalog(set from the config parser under duckdb dialect,e.g.
"testdb"). Db2 is the onlySINGLE_CATALOG_ONLY + UPPERCASEengine,so this case mismatch was never encountered before. This requires a framework-level
fix in
shared.pywhich is outside the scope of this PR.What is included
Core adapter —
sqlmesh/core/engine_adapter/db2.pycatalog_support = SINGLE_CATALOG_ONLY— Db2 databases are fully isolated;cross-database queries are not possible within a single connection
SUPPORTED_DROP_CASCADE_OBJECT_KINDS = []— Db2 raisesSQL0104NonCASCADEMAX_IDENTIFIER_LENGTH = 128set_current_catalog()viaCONNECT TOTIMESTAMPTZstripped toTIMESTAMP(SQL0180N— Db2 rejects UTC offset literals)normalize_identifiersbefore quoting (SQL0204N— CTE alias case mismatch)Connection config —
sqlmesh/core/config/connection.pyDb2ConnectionConfigwithhost,port,database,username,password,db2_schema, and SSL optionsget_catalog()returns.upper()to match Db2 UPPERCASE identifier normalisationFORBIDDEN_STATE_SYNC_ENGINES— Db2 rejects table names startingwith
_; SQLMesh state tables (_snapshots,_environments, etc.) would fail.Users must set
state_connection: duckdbFramework integration
sqlmesh/core/engine_adapter/__init__.py— import guarded byPython >= 3.10 AND find_spec('db2_sqlglot')to prevent a crash in environmentswithout the
db2extra (e.g. dbt 1.6 test runs)sqlmesh/utils/migration.py—db2added toMAX_TEXT_INDEX_LENGTH(255)and
blob_text_type()returnsVARCHAR(32000)CI / infrastructure
.github/workflows/pr.yaml—db2added toengine-tests-dockermatrix.github/scripts/install-prerequisites.sh—db2case installslibxml2-dev build-essential.github/scripts/wait-for-db.sh—db2_ready()probe: port check +db2 connectlooptests/.../docker/compose.db2.yaml— IBM Db2 Community Edition containerMakefile—db2-testtarget;db2added toinstall-devpyproject.toml—db2optional extra (ibm_db+db2-sqlglot-dialect) + pytest markerTests
tests/core/engine_adapter/test_db2.py— 22 unit tests, all passingtests/core/engine_adapter/integration/test_integration_db2.py— 18 Db2-specific integration teststests/core/engine_adapter/integration/__init__.py— SYSCAT comment queries,role-based grant test infrastructure for Db2
tests/core/engine_adapter/integration/config.yaml—inttest_db2gatewaywith
state_connection: duckdbDocumentation
docs/integrations/engines/db2.md— connection options, state connectionguidance, limitations, example config
docs/integrations/overview.md,docs/guides/connections.md,mkdocs.yml— Db2 entries addedChecklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO