feat: support TRIGGER ON UPDATE and SCHEDULE EVERY for MV and ST#1434
feat: support TRIGGER ON UPDATE and SCHEDULE EVERY for MV and ST#1434sd-db wants to merge 4 commits into
Conversation
Adds the full Databricks refresh-schedule grammar on materialized views and streaming tables: SCHEDULE CRON, SCHEDULE EVERY, and TRIGGER ON UPDATE [AT MOST EVERY INTERVAL ...]. - RefreshConfig is a discriminated config with parse-time validation on the new modes. - Idempotent runs no longer emit spurious ALTERs across user/server unit differences. - Auto-REFRESH is suppressed for every/on_update modes. Coverage: - Unit tests for parser, validator, diff, and refresh-schedule macros. - Functional round-trip tests per mode for both MV and ST. - Functional lifecycle tests walking each relation through every mode transition plus a non-refresh-component change.
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||
Drop log-line and SQL-emission assertions, keep only mode-value checks on the resulting relation. Remove tests that only inspected dbt logs.
This comment was marked as duplicate.
This comment was marked as duplicate.
1 similar comment
|
/integration-test |
|
Integration tests dispatched for PR #1434 by @tejassp-db. Track progress in the Actions tab. |
|
Integration results for PR #1434 — UC cluster ❌ cancelled · SQL warehouse ❌ cancelled · All-purpose cluster ❌ failure |
|
/integration-test |
|
Integration tests dispatched for PR #1434 by @sd-db. Track progress in the Actions tab. |
|
Integration results for PR #1434 — UC cluster ⏩ skipped · SQL warehouse ⏩ skipped · All-purpose cluster ⏩ skipped · Shard coverage ⏩ skipped |
|
/integration-test |
|
Integration tests dispatched for PR #1434 by @sd-db. Track progress in the Actions tab. |
|
Integration results for PR #1434 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success |
| ON_UPDATE = "on_update" | ||
|
|
||
|
|
||
| CRON_REGEX = re.compile(r"^CRON '(.*)' AT TIME ZONE '(.*)'$") |
There was a problem hiding this comment.
Do we take into account trailing comments with hash?
There was a problem hiding this comment.
CRON_REGEX is used to parse realtion config from the server side, basically the result from DESCRIBE EXTENDED. The format here is heavily sanitised and quite simple. For CRON_REGEX the value is exiting REGEX as well that is used in production
| CRON_REGEX = re.compile(r"^CRON '(.*)' AT TIME ZONE '(.*)'$") | ||
| EVERY_REGEX = re.compile(r"^EVERY (\d+) (HOURS?|DAYS?|WEEKS?)$", re.IGNORECASE) | ||
| TRIGGER_REGEX = re.compile( | ||
| r"^TRIGGER ON UPDATE(?: AT MOST EVERY INTERVAL (\d+) SECONDS?)?$", |
There was a problem hiding this comment.
It can be any valid time qualifier like MINUTES, HOURS, etc. See docs. Example
CREATE OR REFRESH STREAMING TABLE catalog.schema.customer_orders
TRIGGER ON UPDATE AT MOST EVERY INTERVAL 5 MINUTES
AS SELECT
o.customer_id,
o.name,
o.order_id
FROM catalog.schema.orders o;
There was a problem hiding this comment.
The syntax is for parsing server side values that are returned from DESCIRBE EXTENDED. The syntax is derived from real tests against warehouse to verify on the schema. While in the yml we can have multiple formats on server side it is always stored in seconds
2 participants
Summary
Adds the full Databricks refresh-schedule grammar on materialized views and streaming tables: `SCHEDULE CRON`, `SCHEDULE EVERY`, and `TRIGGER ON UPDATE [AT MOST EVERY INTERVAL ...]`.
Closes #1293.
User-facing config:
Test plan