feat: Replace information schema with describe calls#1432
Open
tejassp-db wants to merge 13 commits into1.12.latestfrom
Open
feat: Replace information schema with describe calls#1432tejassp-db wants to merge 13 commits into1.12.latestfrom
tejassp-db wants to merge 13 commits into1.12.latestfrom
Conversation
Add alternate code path for metadata fetching in _describe_relation methods using DESCRIBE TABLE EXTENDED AS JSON (DBR 17.3+). Replaces 4 information_schema queries in IncrementalTableAPI and get_view_description in MaterializedViewAPI/ViewAPI. Falls back to legacy queries when capability is unavailable. PECOBLR-2546
Wrap long docstrings and expressions to satisfy E501, and add the missing return annotation on DatabricksRelation.is_foreign_table for mypy.
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
3 tasks
Fix primary key and foreign key constraint parser to handle non-delimited identifiers within backticks.
Add functional tests to validate describe json behavior flag true and false.
Collaborator
|
/integration-test |
|
Integration tests dispatched for PR #1432 by @sd-db. Track progress in the Actions tab. |
|
Integration results for PR #1432 — UC cluster ⏩ skipped · SQL warehouse ⏩ skipped · All-purpose cluster ⏩ skipped · Shard coverage ⏩ skipped |
sd-db
added a commit
that referenced
this pull request
May 9, 2026
…ibe-calls Resolves a conflict in MaterializedViewAPI._describe_relation between PR #1432's JSON-metadata gating and the recently-merged fix for #1419 (commit 80e486c, "refresh materialized_view when databricks_tags is set"). The fix added an unconditional fetch_tags call to the MV describe path. PR #1432 restructured the same method behind is_describe_as_json_supported() and dropped that call entirely. Today, DESCRIBE TABLE EXTENDED ... AS JSON does not include tag information, so tags must always be fetched via information_schema -- matching what the PR already does for ViewAPI and IncrementalTableAPI. Resolution: keep the JSON gate for view_description and row_filters, and place fetch_tags outside the gate so MV tag drift is detected on both old and new runtimes. Without this, the #1419 regression silently returns on DBR 17.3+.
…ibe-calls Resolves a conflict in MaterializedViewAPI._describe_relation between PR #1432's JSON-metadata gating and the recently-merged fix for #1419 (commit 80e486c, "refresh materialized_view when databricks_tags is set"). The fix added an unconditional fetch_tags call to the MV describe path. PR #1432 restructured the same method behind is_describe_as_json_supported() and dropped that call entirely. Today, DESCRIBE TABLE EXTENDED ... AS JSON does not include tag information, so tags must always be fetched via information_schema -- matching what the PR already does for ViewAPI and IncrementalTableAPI. Resolution: keep the JSON gate for view_description and row_filters, and place fetch_tags outside the gate so MV tag drift is detected on both old and new runtimes. Without this, the #1419 regression silently returns on DBR 17.3+.
sd-db
force-pushed
the
replace-information-schema-with-describe-calls
branch
from
2ad11df to
7704ea6
Compare
Collaborator
|
/integration-test |
|
Integration tests dispatched for PR #1432 by @sd-db. Track progress in the Actions tab. |
|
Integration results for PR #1432 — UC cluster ✅ success · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success |
Collaborator
|
On the failure, this is known flaky test so can be ignored (I will look to solve for this separately) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace expensive
information_schemaqueries with a singleDESCRIBE TABLE EXTENDED ... AS JSONcall for metadata fetching in_describe_relationmethods. Gated behindDBRCapability.DESCRIBE_TABLE_EXTENDED_AS_JSON(DBR 17.3+), falls back to existing info_schema queries on older runtimes.Changes:
IncrementalTableAPI._describe_relation: replaces 4 info_schema queries (PK, FK, non-null constraints, column masks) with AS JSON parsingMaterializedViewAPI._describe_relation: replacesget_view_description(info_schema.views) with AS JSON parsingViewAPI._describe_relation: same as MVDatabricksDescribeJsonMetadataparser class with composite PK/FK supportis_describe_as_json_supported()gating method (checks HMS, foreign table, capability)is_foreign_tableproperty onDatabricksRelationdescribe_table_extended_as_jsonJinja macroTesting:
is_describe_as_json_supportedunit tests (UC, HMS, foreign table, no capability)Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.PECOBLR-2546