experimental/bundletest: add the cloud backend - #6610
Draft
Sankalp-Mittal wants to merge 15 commits into
Draft
Conversation
Implements the second Backend behind the seam so the same isolation tests run against a real workspace with BUNDLETEST_BACKEND=cloud. deploy/summary/destroy go through the databricks CLI; seed/run/query go through the Databricks SDK. - execute_sql submits to the Statement Execution API, polls to terminal, paginates chunks, and casts each string cell to its manifest column type (row_count() == 2, not "2"); NULLs stay None and it raises on a failed statement. - table_schema reads DESCRIBE TABLE, returning Databricks type spelling. - run_job runs the deployed job (jobs.run_now) and maps result_state/duration. - get_resource returns the rendered `bundle summary` config, hydrating serialized_dashboard/serialized_space from the workspace so source_tables() works for file_path-only dashboards and genie spaces. - volumes: put_file/read_volume_file resolve the /Volumes/<resource>/<path> shorthand to the real UC path and parse csv/json/parquet with duckdb. The SDK is an optional `cloud` extra, lazily imported, so a local-only install never pulls it in. Adds unit tests for the pure logic (casting, literal/type rendering, namespace discovery, volume-path resolution) that run without a workspace. Co-authored-by: Isaac <no-reply@databricks.com>
…dles' into sankalp-mittal/dabs-cloud-backend
A successful DDL/DML statement (CREATE/INSERT/DROP, and the schema preparation that seeding does) returns SUCCEEDED with result=None and no manifest, so reading result.data_array raised AttributeError — on the very first real seed. Return an empty list in that case, and pin it with a result=None test case the fake missed. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Waiting for approvalCould not determine reviewers from git history. Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
Sankalp-Mittal
added this pull request to stack #6604
September 10, 2026 15:31
Sankalp-Mittal
marked this pull request as draft
September 10, 2026 15:31
Live validation against a real warehouse showed a doubled single quote ('') does
NOT escape a quote in Databricks/Spark SQL — it drops it ('a''b' -> ab). Spark
escapes string literals with a backslash, so seed values with an embedded quote
were silently corrupted. Escape both the backslash and the single quote with a
backslash instead (verified end-to-end: 'a''b'->ab vs 'a\'b'->a'b on the warehouse).
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 51f333b
Top 3 slowest tests (at least 2 minutes):
|
The local gallery (examples/orders_bundle) declares one of every resource kind for static-config reading and is deliberately NOT deployable (fake external locations, metastore-admin catalog, nonexistent models). The cloud backend really deploys, so it needs a deployable fixture: two SQL jobs, a managed volume, and a file_path dashboard, all under main.bundletest_cloud. The module-scoped fixture creates that schema, deploys, and drops it CASCADE afterwards; its tests are collected only when BUNDLETEST_BACKEND=cloud. Validated end-to-end against a real workspace (azure-dogfood): deploy, run_job on both deployed jobs, execute_sql/table_schema round-trips, get_resource off `bundle summary` (job id, volume catalog/schema/name, and the file_path dashboard's serialized_dashboard, which the summary inlines), source_tables, and volume upload/read — 5 passed, clean teardown. Co-authored-by: Isaac <no-reply@databricks.com>
…source Live deploy showed `bundle summary -o json` already inlines serialized_dashboard for a file_path dashboard, and the config mutators confirm the same for serialized_space (configure_dashboards_serialized_dashboard / configure_genie_space_serialized_space read the file_path at load time). So the `if "serialized_dashboard" not in cfg` hydration branches could never fire — get_resource just returns the rendered summary config, which already carries the serialized form source_tables() needs. Removes the dead lakeview.get / genie.get_space branches and the _deployed_id helper. Verified: the cloud E2E dashboard source_tables test (file_path source) still passes. Co-authored-by: Isaac <no-reply@databricks.com>
Point the "Run it on cloud" section at the examples/cloud_orders fixture and the actual BUNDLETEST_BACKEND=cloud invocation. Co-authored-by: Isaac <no-reply@databricks.com>
Cover run_job result mapping (SUCCESS/FAILED, state_message, run_duration ms->seconds), table_schema DESCRIBE row filtering (blank / '#'-prefixed), seed_table CREATE+INSERT with inferred types and Spark-escaped literals + teardown tracking, and best-effort teardown. All via the fake-client pattern, no workspace. 22 tests in the file, full suite 73 passed. Co-authored-by: Isaac <no-reply@databricks.com>
From an independent review of the diff: - read_volume_file: write the downloaded file into a temp dir and read it via a forward-slash path with an explicitly-closed connection, instead of handing a still-open NamedTemporaryFile to DuckDB — the latter fails on Windows (can't reopen an open temp file; backslash path breaks the SQL literal). Matches the local backend's approach. - seed_table: backtick-quote column identifiers so a reserved word or special char works, matching the local backend's quoting. - _ensure_job_schemas: skip a non-file sql_task and make CREATE SCHEMA best-effort, so a query/alert task or a misparsed struct reference can't abort run_job. Adds a read_volume_file unit test; kept DECIMAL->float deliberately (float-literal equality in assertions). Local suite 74 passed; cloud E2E re-verified 5 passed on azure-dogfood. Co-authored-by: Isaac <no-reply@databricks.com>
Running the whole suite with BUNDLETEST_BACKEND=cloud errored on everything because two collections aren't cloud things: - examples/orders_bundle is the local static-config gallery (one of every resource kind, several not deployable) — it's read, never deployed. Skip its collection on cloud, the mirror of examples/cloud_orders skipping on local. The cloud fixture is examples/cloud_orders. - tests/test_assertions.py exercises the local backend's SQL/routing behavior with throwaway bundles that have no `bundle:` name; pin its bundle_env calls to backend="local" so the env var can't send them to a cloud deploy. Now `BUNDLETEST_BACKEND=cloud ... pytest` runs only the cloud-appropriate tests (40 passed); local is unchanged (74 passed, 1 skipped). Co-authored-by: Isaac <no-reply@databricks.com>
… test get_resource no longer reads the serialized form back from the workspace — `bundle summary` inlines a file_path dashboard's serialized_dashboard at config-load. Update the module docstring and rename test_dashboard_serialized_is_hydrated_from_file_path -> test_dashboard_source_tables_from_file_path so the comments describe the actual mechanism. Co-authored-by: Isaac <no-reply@databricks.com>
… validation get_resource returns the DECLARED config (bundle summary) — the shared-seam shape every resource handle depends on. It can't show what the server filled in or normalized, which is part of validating a real deployment. Add a separate, cloud-only CloudBackend.get_deployed( kind, name) that reads the resource back from the workspace via the SDK and returns the raw server object as a dict. It's additive and cloud-only (not in the Backend protocol, no lossy shape-normalization, get_resource untouched), so tests that use it must be @cloud_only. Scoped (YAGNI) to the kinds validated live against examples/cloud_orders: jobs, dashboards, volumes; other kinds raise a clear error. Live-verified on azure-dogfood: a deployed job carries server-filled fields absent from databricks.yml (settings.format=MULTI_TASK, max_concurrent_runs=1, run_as_user_name). Cloud E2E 6 passed; local 76 passed, 1 skipped. Design agreed with the base/seam owner (separate accessor over routing through get_resource). Co-authored-by: Isaac <no-reply@databricks.com>
1. Isolation: the cloud fixture now uses a UNIQUE per-run schema (main.bundletest_cloud_<run>) and bundle name, deployed from a per-run copy with the placeholder substituted (the .sql / .lvdash.json artifacts hardcode the schema and DABs doesn't interpolate file contents). Two concurrent runs no longer collide, and the schema is still created up front + dropped CASCADE. Tests take a `schema` fixture instead of a fixed constant. 2. Cleanup no longer masks failures: teardown logs a warning on a failed drop/`bundle destroy` (and the fixture warns on a failed schema drop) so leaked, billable resources aren't hidden — still non-raising, still doesn't mask the primary test failure. 3. Narrowed broad excepts: read_volume_file catches only databricks.sdk.errors.NotFound (so auth/permission errors surface instead of masquerading as a missing file); _ensure_job_schemas catches only the RuntimeError execute_sql raises for a statement failure (auth errors propagate). Local 76 passed, 1 skipped; ruff clean. Co-authored-by: Isaac <no-reply@databricks.com>
…dles' into sankalp-mittal/dabs-cloud-backend
…dles' into sankalp-mittal/dabs-cloud-backend # Conflicts: # experimental/bundletest/README.md # experimental/bundletest/tests/test_assertions.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the cloud backend for
experimental/bundletest— a second implementation behind theBackendseam so the same isolation tests run against a real Databricks workspace withBUNDLETEST_BACKEND=cloud. Where the local DuckDB backend simulates, this deploys the bundle for real and drives the workspace, so thecloud_onlyassertions (Databricks type names, SLA timing, notebook/Python jobs) actually run instead of skipping.Third PR in the stack: base
dabs-testing-framework(#6597) ← resource handles (#6603) ← this.How
databricks bundle deploy/summary/destroyvia subprocess for bundle ops. The SDK is an optionalcloudextra, lazily imported, so a local-only install never pulls it in.execute_sqlsubmits to the Statement Execution API, polls to terminal, paginates result chunks, and casts each cell to its manifest column type — the API returns every value as a string, sorow_count() == 2, not"2". NULLs stayNone, a no-result-set DDL/DML returns[], and a failed statement raises.table_schemareadsDESCRIBE TABLE, returning the Databricks type spelling (decimal(10,2)).run_jobruns the deployed job viajobs.run_now, mappingRunResultState/duration (ms→s).get_resourcereturns the renderedbundle summary -o jsonconfig. The config mutators inlineserialized_dashboard/serialized_spacefrom afile_pathat load time, sosource_tables()gets the serialized form straight from the summary — no separate workspace read./Volumes/<resource>/<path>shorthand to the real UC path and parse csv/json/parquet with duckdb.seed_tableescapes string literals the Spark way (backslash, since a doubled''drops the quote on Spark — caught during live validation).BUNDLETEST_PROFILE,BUNDLETEST_WAREHOUSE_ID,BUNDLETEST_TARGET; bundle variables the normal DABs way (BUNDLE_VAR_*), sodeploystays bundle-agnostic.Testing
examples/cloud_orders/(two SQL jobs + a managed volume + a file_path dashboard, undermain.bundletest_cloud; collected only whenBUNDLETEST_BACKEND=cloud; schema created up front and droppedCASCADEafter). Validated end-to-end against a real workspace: deploy →run_jobon both jobs →execute_sql/table_schema(incl.decimal(10,2)) →get_resource→ dashboardsource_tables()from a file_path source → volume upload/read — 5 passed, clean teardown.examples/orders_bundle/remains the local static-config gallery (one of every resource kind, read not deployed) — deliberately not cloud-deployable, which is why the cloud tier gets its own trimmed deployable fixture.uv run --extra dev pytest); ruff clean.This pull request and its description were written by Isaac.