Skip to content

feat: Add IBM Db2 engine adapter - #6030

Open
hdm-db2-eco-system wants to merge 1 commit into
SQLMesh:mainfrom
hdm-db2-eco-system:db2-adapter-pr
Open

feat: Add IBM Db2 engine adapter#6030
hdm-db2-eco-system wants to merge 1 commit into
SQLMesh:mainfrom
hdm-db2-eco-system:db2-adapter-pr

Conversation

@hdm-db2-eco-system

Copy link
Copy Markdown

Adds a IBM Db2 engine adapter for SQLMesh, including connection
configuration, Docker-based CI, integration tests, and engine-specific documentation.

CI results (stable baseline — Docker Db2 Community Edition)

81 passed · 46 skipped · 0 failed

Tested against: icr.io/db2_community/db2:latest (Db2 11.5)


What works

Model kinds

Kind Status
FULL ✅ Working
INCREMENTAL_BY_TIME_RANGE ✅ Working
INCREMENTAL_BY_UNIQUE_KEY ✅ Working — MERGE INTO confirmed
INCREMENTAL_BY_PARTITION ✅ Working
INCREMENTAL_UNMANAGED ✅ Working
VIEW ✅ Working
SEED ✅ Working
SCD_TYPE_2_BY_TIME ⏭ Deferred — see below
SCD_TYPE_2_BY_COLUMN ⏭ Deferred — see below

Core engine operations

Operation Notes
CREATE TABLE Primary key columns get NOT NULL automatically
CTAS Appends mandatory WITH DATA clause
MERGE INTO TARGET/SOURCE alias substitution
CREATE INDEX Idempotent — checks SYSCAT.INDEXES first
CREATE/DROP SCHEMA Existence checked via SYSCAT.SCHEMATA
DROP VIEW Existence checked via SYSCAT.VIEWS
GRANT/REVOKE Uses SYSCAT.TABAUTH (Db2 has no INFORMATION_SCHEMA.TABLE_PRIVILEGES)
TRUNCATE Implemented as DELETE FROM (Db2 has no native TRUNCATE)
Table/column comments COMMENT ON TABLE / COMMENT ON COLUMN
Metadata queries All via SYSCAT catalog views

What is skipped and why

All skips are in tests/core/engine_adapter/integration/test_integration.py
with documented pytest.skip() reasons. Search for if ctx.dialect == "db2"
in that file to see every skip block.

1. SCD Type 2 — 4 tests skipped

Tests: test_scd_type_2_by_time, test_scd_type_2_by_time_source_columns,
test_scd_type_2_by_column, test_scd_type_2_by_column_source_columns

Error: SQL20521N reason 7 — Db2's SQL preprocessor treats identifiers
starting with _ as conditional compilation directives.

Root cause: SQLMesh's _scd_type_2() in base.py generates _exists,
_key0, _row_number, _t as column aliases — all underscore-prefixed.
The SCD staging table also uses CTAS which Db2 does not support natively.


2. View comments — 3 tests skipped

Tests: test_create_view, test_create_view_source_columns,
test_get_data_objects

Error: SQL0104N — Db2 has no COMMENT ON VIEW statement.


3. test_sushi — 1 test skipped

Error: SQL0104NCREATE SCHEMA IF NOT EXISTS is not valid Db2 SQL.
The before_all statements in the sushi example project emit IF NOT EXISTS
unconditionally through the db2-sqlglot-dialect generator.


4. ctx.create_context() tests — 6 tests skipped

Tests: test_batch_size_on_incremental_by_unique_key_model,
test_state_migrate_from_scratch, test_python_model_column_order,
test_unicode_characters, test_grants_plan,
test_incremental_by_unique_key_model_when_matched

Error: SQLMeshError: Cannot create a snapshot in a different catalog

Root cause: shared.py:346 performs a case-sensitive == comparison
between catalog_name (uppercased by Db2 normalisation, e.g. "TESTDB")
and _default_catalog (set from the config parser under duckdb dialect,
e.g. "testdb"). Db2 is the only SINGLE_CATALOG_ONLY + UPPERCASE engine,
so this case mismatch was never encountered before. This requires a framework-level
fix in shared.py which is outside the scope of this PR.


What is included

Core adapter — sqlmesh/core/engine_adapter/db2.py

  • catalog_support = SINGLE_CATALOG_ONLY — Db2 databases are fully isolated;
    cross-database queries are not possible within a single connection
  • SUPPORTED_DROP_CASCADE_OBJECT_KINDS = [] — Db2 raises SQL0104N on CASCADE
  • MAX_IDENTIFIER_LENGTH = 128
  • set_current_catalog() via CONNECT TO
  • TIMESTAMPTZ stripped to TIMESTAMP (SQL0180N — Db2 rejects UTC offset literals)
  • normalize_identifiers before quoting (SQL0204N — CTE alias case mismatch)

Connection config — sqlmesh/core/config/connection.py

  • Db2ConnectionConfig with host, port, database, username, password,
    db2_schema, and SSL options
  • get_catalog() returns .upper() to match Db2 UPPERCASE identifier normalisation
  • Excluded from FORBIDDEN_STATE_SYNC_ENGINES — Db2 rejects table names starting
    with _; SQLMesh state tables (_snapshots, _environments, etc.) would fail.
    Users must set state_connection: duckdb

Framework integration

  • sqlmesh/core/engine_adapter/__init__.py — import guarded by
    Python >= 3.10 AND find_spec('db2_sqlglot') to prevent a crash in environments
    without the db2 extra (e.g. dbt 1.6 test runs)
  • sqlmesh/utils/migration.pydb2 added to MAX_TEXT_INDEX_LENGTH (255)
    and blob_text_type() returns VARCHAR(32000)

CI / infrastructure

  • .github/workflows/pr.yamldb2 added to engine-tests-docker matrix
  • .github/scripts/install-prerequisites.shdb2 case installs libxml2-dev build-essential
  • .github/scripts/wait-for-db.shdb2_ready() probe: port check + db2 connect loop
  • tests/.../docker/compose.db2.yaml — IBM Db2 Community Edition container
  • Makefiledb2-test target; db2 added to install-dev
  • pyproject.tomldb2 optional extra (ibm_db + db2-sqlglot-dialect) + pytest marker

Tests

  • tests/core/engine_adapter/test_db2.py — 22 unit tests, all passing
  • tests/core/engine_adapter/integration/test_integration_db2.py — 18 Db2-specific integration tests
  • tests/core/engine_adapter/integration/__init__.py — SYSCAT comment queries,
    role-based grant test infrastructure for Db2
  • tests/core/engine_adapter/integration/config.yamlinttest_db2 gateway
    with state_connection: duckdb

Documentation

  • docs/integrations/engines/db2.md — connection options, state connection
    guidance, limitations, example config
  • docs/integrations/overview.md, docs/guides/connections.md, mkdocs.yml — Db2 entries added

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Adds a complete IBM Db2 LUW engine adapter for SQLMesh, including
connection configuration, Docker-based CI, integration tests, and
engine-specific documentation.

## What is included

### Core adapter (sqlmesh/core/engine_adapter/db2.py)
- Db2EngineAdapter implementing all standard SQLMesh engine operations
- SYSCAT-based metadata queries (columns, tables, schemas, indexes, views)
- CTAS with mandatory WITH DATA clause
- MERGE INTO with TARGET/SOURCE alias substitution
- CREATE INDEX with SYSCAT.INDEXES existence check
- CREATE/DROP SCHEMA using SYSCAT.SCHEMATA
- DROP VIEW using SYSCAT.VIEWS existence check
- GRANT/REVOKE using SYSCAT.TABAUTH (not INFORMATION_SCHEMA)
- Truncate implemented as DELETE FROM (Db2 has no TRUNCATE)
- TIMESTAMPTZ stripped to TIMESTAMP (SQL0180N fix)
- normalize_identifiers before quoting (SQL0204N fix)
- catalog_support = SINGLE_CATALOG_ONLY (Db2 databases are isolated)
- SUPPORTED_DROP_CASCADE_OBJECT_KINDS = [] (SQL0104N fix)
- MAX_IDENTIFIER_LENGTH = 128
- set_current_catalog via CONNECT TO

### Connection config (sqlmesh/core/config/connection.py)
- Db2ConnectionConfig with host/port/database/username/password/db2_schema
- SSL options (ssl, ssl_cert, ssl_key, ssl_ca)
- get_catalog() returns .upper() to match Db2 UPPERCASE normalisation
- Excluded from FORBIDDEN_STATE_SYNC_ENGINES with explanation comment
  (Db2 rejects table names starting with underscore)

### Framework integration
- sqlmesh/core/engine_adapter/__init__.py: conditional import guarded
  by Python >= 3.10 AND find_spec('db2_sqlglot') — prevents import
  crash in environments without the db2 extra installed
- sqlmesh/utils/migration.py: db2 added to MAX_TEXT_INDEX_LENGTH
  (255) and blob_text_type() returns VARCHAR(32000)

## CI / infrastructure
- .github/workflows/pr.yaml: db2 added to engine-tests-docker matrix
- .github/scripts/install-prerequisites.sh: db2 installs libxml2-dev
- .github/scripts/wait-for-db.sh: db2_ready() readiness probe
- tests/.../docker/compose.db2.yaml: IBM Db2 Community Edition image
- Makefile: db2-test target with junitxml; db2 added to install-dev
- pyproject.toml: db2 optional extra + pytest marker

## Tests
- tests/core/engine_adapter/test_db2.py: 22 unit tests (all passing)
- tests/core/engine_adapter/integration/test_integration_db2.py:
  18 Db2-specific integration tests
- tests/core/engine_adapter/integration/__init__.py: SYSCAT comment
  queries, role-based grant infrastructure for Db2
- tests/core/engine_adapter/integration/config.yaml: inttest_db2
  gateway with DuckDB state_connection
- tests/core/engine_adapter/integration/test_integration.py:
  skip blocks with documented reasons for 20 tests; adaptations
  for uppercase identifiers, TIMESTAMPTZ, grants

## CI results (stable baseline on Docker Db2 Community Edition)
- 81 passed · 46 skipped · 0 failed

## Known limitations and skipped tests

### SCD Type 2 (4 tests skipped)
Db2 SQL preprocessor treats identifiers starting with '_' as
conditional compilation directives (SQL20521N reason 7). SQLMesh
generates _exists, _key0, _row_number, _t as aliases. Additionally
SCD staging uses CTAS which Db2 does not support natively.
Fix: override _scd_type_2() in Db2EngineAdapter to rename aliases.

### View comments (3 tests skipped)
Db2 has no COMMENT ON VIEW statement (SQL0104N).

### test_sushi (1 test skipped)
CREATE SCHEMA IF NOT EXISTS is not valid Db2 SQL (SQL0104N).
Fix: add create_sql() override to db2-sqlglot-dialect to strip
IF NOT EXISTS from CREATE SCHEMA.

### ctx.create_context() tests (6 tests skipped)
shared.py:346 uses a case-sensitive == comparison between
catalog_name (TESTDB, Db2 uppercase) and _default_catalog
(testdb, duckdb-dialect lowercase). This is a one-line upstream
fix in shared.py that cannot be made in this PR:
  catalog_name.upper() != (engine_adapter._default_catalog or '').upper()

## Documentation
- docs/integrations/engines/db2.md: connection options, state
  connection guidance, limitations, example config
- docs/integrations/overview.md: Db2 entry added
- docs/guides/connections.md: Db2 link added
- mkdocs.yml: db2.md nav entry added

Signed-off-by: IBM Db2 Eco System <Hdm-dev-persona-db2-eco-system@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant