direct: report why a migration failed in telemetry

acceptance/bundle/migrate/auto-migrate-conversion-failure/databricks.yml

@@ -0,0 +1,13 @@
+bundle:
+  name: test-bundle
+
+resources:
+  secret_scopes:
+    my_scope:
+      name: my-scope
+      permissions:
+        # An unknown level: SecretScopeFixups rejects it when it prepares the
+        # config for the direct engine, so the state conversion fails while the
+        # terraform deploy itself succeeds.
+        - level: BOGUS
+          user_name: someone@example.com

acceptance/bundle/migrate/auto-migrate-conversion-failure/output.txt

@@ -0,0 +1,46 @@
+
+=== Not opted in: the conversion failure is reported as a failed dry run
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Warning: invalid value "BOGUS" for enum field. Valid values are [READ WRITE MANAGE]
+  at resources.secret_scopes.my_scope.permissions[0].level
+
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created secret_scopes.my_scope
+Created secret_scopes.my_scope.permissions
+Files: 4 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+Warn: post-deploy dry-run migration to direct: unknown permission level "BOGUS" for secret scope
+Warn: The warnings above are from a dry-run migration to the direct deployment engine (https://docs.databricks.com/aws/en/dev-tools/bundles/direct).
+Your deployment is not affected and works normally, but you may experience these issues when migrating to the direct deployment engine.
+Please forward these warnings to dabs-feedback@databricks.com
+
+>>> print_migration_telemetry
+direct_drymigrate_success false
+direct_drymigrate_warnings false
+direct_migrate_safe_error unknown permission level %q for secret scope
+
+=== Opted in: the same failure stops the automatic migration
+>>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
+Warning: invalid value "BOGUS" for enum field. Valid values are [READ WRITE MANAGE]
+  at resources.secret_scopes.my_scope.permissions[0].level
+
+Warn: Direct engine selected via DATABRICKS_BUNDLE_ENGINE environment variable but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 2 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
+Warn: post-deploy dry-run migration to direct: unknown permission level "BOGUS" for secret scope
+Warn: The warnings above are from a dry-run migration to the direct deployment engine (https://docs.databricks.com/aws/en/dev-tools/bundles/direct).
+Your deployment is not affected and works normally, but you may experience these issues when migrating to the direct deployment engine.
+Please forward these warnings to dabs-feedback@databricks.com
+Warn: Direct engine was selected but the migration reported issues; automatic migration to the direct deployment engine is stopped. Address the issues above or run "databricks bundle deployment migrate" manually.
+
+>>> print_migration_telemetry
+direct_migrate_error true
+direct_migrate_safe_error unknown permission level %q for secret scope
+
+=== State is still terraform, so nothing was migrated
+
+>>> find .databricks/bundle -name resources.json -type f
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate

acceptance/bundle/migrate/auto-migrate-conversion-failure/script

@@ -0,0 +1,20 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+# An unknown permission level is only a warning for the terraform deploy, but
+# SecretScopeFixups rejects it while preparing the config for the direct engine.
+# That is a state conversion failure: no API call is involved, so unlike the
+# commit failures it cannot be produced by injecting a fault.
+
+title "Not opted in: the conversion failure is reported as a failed dry run"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt
+
+title "Opted in: the same failure stops the automatic migration"
+trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt
+
+title "State is still terraform, so nothing was migrated\n"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f

acceptance/bundle/migrate/auto-migrate-push-failure/output.txt

@@ -21,6 +21,7 @@ Warn: automatic migration to direct engine failed: pushing direct state to works
 
 >>> print_migration_telemetry
 direct_migrate_commit_error true
+direct_migrate_commit_safe_error pushing direct state to workspace: access denied [403 INJECTED]
 
 === Local state was NOT rewritten (still terraform)
 

acceptance/bundle/migrate/auto-migrate-secret-scope/databricks.yml

@@ -0,0 +1,10 @@
+bundle:
+  name: test-bundle
+
+resources:
+  secret_scopes:
+    my_scope:
+      name: my-scope
+      permissions:
+        - level: READ
+          user_name: someone@example.com

acceptance/bundle/migrate/auto-migrate-secret-scope/output.txt

@@ -0,0 +1,41 @@
+
+=== Initial deploy uses the terraform engine
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created secret_scopes.my_scope
+Created secret_scopes.my_scope.permissions
+Files: 5 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Opt in to the direct engine and redeploy, which migrates the state
+>>> update_file.py databricks.yml name: test-bundle name: test-bundle
+  engine: direct
+
+>>> [CLI] bundle deploy
+Warn: Direct engine selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11 but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 3 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
+Migrating state to direct deployment engine (selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11)...
+Migrated 2 resources to direct deployment engine.
+
+>>> print_migration_telemetry
+direct_migrated_via_config true
+
+=== The migrated state records the MANAGE ACL the fixups added
+
+>>> print_state.py
+[
+  {
+    "permission": "READ",
+    "principal": "someone@example.com"
+  },
+  {
+    "permission": "MANAGE",
+    "principal": "[USERNAME]"
+  }
+]
+
+=== So the first plan after migrating is a no-op
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged

acceptance/bundle/migrate/auto-migrate-secret-scope/script

@@ -0,0 +1,25 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+# SecretScopeFixups adds a MANAGE ACL for the current user, which the backend
+# creates implicitly. The fixups mutate typed config while the state is built
+# from the dynamic tree, so the migration has to sync one into the other;
+# without that the ACL is missing from the state below. The plan is a no-op
+# either way, because the acls resource re-reads the live ACLs.
+
+title "Initial deploy uses the terraform engine"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+rm -f out.requests.txt
+
+title "Opt in to the direct engine and redeploy, which migrates the state"
+trace update_file.py databricks.yml "name: test-bundle" $'name: test-bundle\n  engine: direct'
+trace $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt
+
+title "The migrated state records the MANAGE ACL the fixups added\n"
+trace print_state.py | jq '.state["resources.secret_scopes.my_scope.permissions"].state.acls'
+
+title "So the first plan after migrating is a no-op"
+trace $CLI bundle plan | contains.py "2 unchanged"
+
+rm -f out.requests.txt

acceptance/bundle/migrate/auto-migrate-secret-scope/test.toml

@@ -0,0 +1,3 @@
+Cloud = false
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
+EnvMatrix.DMS = [""]

acceptance/bundle/migrate/auto-migrate-tfbackup-failure/output.txt

@@ -21,6 +21,7 @@ Warn: automatic migration to direct engine failed: pushing direct state to works
 
 >>> print_migration_telemetry
 direct_migrate_commit_error true
+direct_migrate_commit_safe_error pushing direct state to workspace: deleting remote terraform state: %w [403 INJECTED]
 
 === Local state was NOT rewritten (still terraform)
 

acceptance/bundle/migrate/reference-dabs-only-field/databricks.yml

@@ -0,0 +1,24 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    src:
+      name: source
+      tasks:
+        - task_key: t
+          new_cluster: &cluster
+            spark_version: 15.4.x-scala2.12
+            node_type_id: Standard_DS3_v2
+            num_workers: 1
+            # autotermination_minutes has no Terraform equivalent for a job
+            # cluster, so it is absent from terraform.tfstate. Terraform never
+            # sees this reference either: the field is dropped on conversion.
+            autotermination_minutes: 20
+    dst:
+      name: dst
+      tasks:
+        - task_key: t
+          new_cluster:
+            <<: *cluster
+            autotermination_minutes: ${resources.jobs.src.tasks[0].new_cluster.autotermination_minutes}

acceptance/bundle/migrate/reference-dabs-only-field/output.txt

@@ -0,0 +1,59 @@
+
+=== Deploy on terraform: the dry run already reports the failure
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Warning: unknown field: autotermination_minutes
+  at task[0].new_cluster
+  in databricks.yml:17:13
+
+Warning: unknown field: autotermination_minutes
+  at task[0].new_cluster
+  in databricks.yml:17:13
+
+Created jobs.dst
+Created jobs.src
+Files: 4 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+Warn: post-deploy dry-run migration to direct: resources.jobs.dst: cannot resolve field "tasks[0].new_cluster.autotermination_minutes" (template "${resources.jobs.src.tasks[0].new_cluster.autotermination_minutes}"): jobs.dst field tasks[0].new_cluster.autotermination_minutes: method A: jobs: "tasks[0].new_cluster.autotermination_minutes" is a DABs-only field with no Terraform equivalent; method B: cannot look up "resources.jobs.src.tasks[0].new_cluster.autotermination_minutes": jobs: "tasks[0].new_cluster.autotermination_minutes" is a DABs-only field with no Terraform equivalent
+Warn: The warnings above are from a dry-run migration to the direct deployment engine (https://docs.databricks.com/aws/en/dev-tools/bundles/direct).
+Your deployment is not affected and works normally, but you may experience these issues when migrating to the direct deployment engine.
+Please forward these warnings to dabs-feedback@databricks.com
+
+>>> print_migration_telemetry
+direct_drymigrate_success false
+direct_drymigrate_warnings false
+direct_migrate_safe_error jobs.*: cannot resolve field %q (template %q): jobs.%s field %s: method A: %w; method B: cannot look up %q: %w
+
+>>> update_file.py databricks.yml name: test-bundle name: test-bundle
+  engine: direct
+
+=== Opt in via config: the same failure stops the automatic migration
+>>> [CLI] bundle deploy
+Warn: Direct engine selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11 but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Warning: unknown field: autotermination_minutes
+  at task[0].new_cluster
+  in databricks.yml:18:13
+
+Warning: unknown field: autotermination_minutes
+  at task[0].new_cluster
+  in databricks.yml:18:13
+
+Files: 3 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
+Warn: post-deploy dry-run migration to direct: resources.jobs.dst: cannot resolve field "tasks[0].new_cluster.autotermination_minutes" (template "${resources.jobs.src.tasks[0].new_cluster.autotermination_minutes}"): jobs.dst field tasks[0].new_cluster.autotermination_minutes: method A: jobs: "tasks[0].new_cluster.autotermination_minutes" is a DABs-only field with no Terraform equivalent; method B: cannot look up "resources.jobs.src.tasks[0].new_cluster.autotermination_minutes": jobs: "tasks[0].new_cluster.autotermination_minutes" is a DABs-only field with no Terraform equivalent
+Warn: The warnings above are from a dry-run migration to the direct deployment engine (https://docs.databricks.com/aws/en/dev-tools/bundles/direct).
+Your deployment is not affected and works normally, but you may experience these issues when migrating to the direct deployment engine.
+Please forward these warnings to dabs-feedback@databricks.com
+Warn: Direct engine was selected but the migration reported issues; automatic migration to the direct deployment engine is stopped. Address the issues above or run "databricks bundle deployment migrate" manually.
+
+>>> print_migration_telemetry
+direct_migrate_error true
+direct_migrate_safe_error jobs.*: cannot resolve field %q (template %q): jobs.%s field %s: method A: %w; method B: cannot look up %q: %w
+
+=== State is still terraform, so nothing was migrated
+
+>>> find .databricks/bundle -name resources.json -type f
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate

acceptance/bundle/migrate/reference-dabs-only-field/script

@@ -0,0 +1,24 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+# A DABs-only field pointing at another resource's DABs-only field. Neither is in
+# terraform.tfstate, so both of the conversion's resolution methods fail.
+#
+# The deploy is unaffected: the field has no Terraform equivalent, so it is
+# dropped on conversion and Terraform never sees the reference inside it. That is
+# what lets the conversion, rather than the deploy, be the thing that fails.
+title "Deploy on terraform: the dry run already reports the failure"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt
+
+trace update_file.py databricks.yml "name: test-bundle" "name: test-bundle
+  engine: direct"
+
+title "Opt in via config: the same failure stops the automatic migration"
+trace $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt
+
+title "State is still terraform, so nothing was migrated\n"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f

acceptance/bundle/migrate/reference-direct-only-resource/app/app.py

@@ -0,0 +1 @@
+placeholder

acceptance/bundle/migrate/reference-direct-only-resource/databricks.yml

@@ -0,0 +1,13 @@
+bundle:
+  name: test-bundle
+
+resources:
+  #catalogs: {my_catalog: {name: my-catalog}}
+  apps:
+    my_app:
+      name: my-app
+      source_code_path: ./app
+      config:
+        env:
+          - name: CATALOG
+            value: plain

acceptance/bundle/migrate/reference-direct-only-resource/output.txt

@@ -0,0 +1,22 @@
+
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created apps.my_app
+Files: 5 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> update_file.py databricks.yml #catalogs catalogs
+
+>>> update_file.py databricks.yml value: plain value: ${resources.catalogs.my_catalog.name}
+
+=== Direct requested: the catalog is skipped by terraform and the migration still succeeds
+>>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
+Warn: Direct engine selected via DATABRICKS_BUNDLE_ENGINE environment variable but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 3 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
+Migrating state to direct deployment engine (selected via DATABRICKS_BUNDLE_ENGINE environment variable)...
+Migrated 1 resource to direct deployment engine.
+
+>>> print_migration_telemetry
+direct_migrated_via_env true

acceptance/bundle/migrate/reference-direct-only-resource/script

@@ -0,0 +1,18 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+# Recording that a reference into a direct-only resource type does not break the
+# migration. Terraform cannot deploy catalogs at all, so the reference resolves
+# to nothing in terraform.tfstate — but apps.config is inline app.yaml rather
+# than an API field, so ExtractReferences drops the reference before it is ever
+# resolved. See bundle/direct/bundle_plan.go: refs are kept only for fields that
+# exist in the state type.
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+rm -f out.requests.txt
+
+trace update_file.py databricks.yml "#catalogs" "catalogs"
+trace update_file.py databricks.yml "value: plain" 'value: ${resources.catalogs.my_catalog.name}'
+
+title "Direct requested: the catalog is skipped by terraform and the migration still succeeds"
+trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy
+trace print_migration_telemetry
+rm -f out.requests.txt

acceptance/bundle/migrate/reference-methods-disagree/databricks.yml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+# Enabled by the script before the second deploy, so the first one leaves a state
+# the two resolution methods agree on.
+#experimental: {scripts: {postdeploy: "python3 drift_stored_name.py"}}
+
+resources:
+  jobs:
+    src:
+      name: source
+    dst:
+      # Method A reads dst's own stored name; Method B evaluates this template
+      # against src's. The post-deploy script makes those two differ.
+      name: ${resources.jobs.src.name}