How to set up CI/CD for Azure Databricks

What belongs on each side:

And what actually lives in each layer:

• Azure foundation: resource groups, VNets, private endpoints, storage accounts, Key Vault, managed identities, diagnostic settings and the workspace itself.
• Databricks platform: workspaces, the Unity Catalog metastore, workspace assignment, groups, service principals, cluster policies, SQL warehouses, external locations, storage credentials and permissions.
• Analytics workload: jobs, Lakeflow pipelines, notebooks, wheels, libraries, dashboards, model serving endpoints, variables and targets.
• Release orchestration: validation, tests, terraform plan and apply, bundle validate, deploy and run, and the approvals in between.

Terraform can manage most Databricks resources, but it carries state while doing so. Every notebook change you pull into it becomes a state change that has to pass through the platform team. A bundle is meant to work the other way around: a complete project definition with source files, resource definitions, tests and deployment configuration in one version.

Two older approaches still exist and both share the same limitation. With Git folders only, your code is in source control but your job and pipeline configuration is not. With Git plus jobs, the job pulls code from Git at run time, but task order, compute and schedule stay outside source control. For deployment across multiple workspaces that falls short.

Unity Catalog cuts straight through this. Workspaces in the same region share one metastore, so your permission model and your lineage live in one place. You draw the line between environments inside it with workspace-catalog binding: that ties a production catalog to production workspaces, even when somebody elsewhere holds explicit grants.

Identity belongs at account level, should come from your identity provider and should sit in groups. Give ownership of production objects to a group and let jobs run under a service principal. A job that runs as a person falls over the moment that person leaves.

Secrets do not belong in notebooks, pipelines or YAML. Use Databricks secrets, optionally with a Key Vault-backed scope. That scope is read-only from Databricks and uses the Key Vault access policy model, not Azure RBAC. Cut scopes along roles or applications, not along people.

infra/
  modules/
    azure-databricks-workspace/
    unity-catalog/
    networking/
    identity/
    policies/
  envs/
    dev/
      backend.tf
      main.tf
      terraform.tfvars
    test/
    acc/
    prod/

Put your state in Azure Storage rather than locally. Local state does not collaborate, can hold sensitive values and is one bad cleanup away from gone. Remote state gives you central storage, locking and encryption. Use a separate state key per environment.

The order inside Terraform is fixed, because the second provider needs the first one. The AzureRM provider creates the resource group and the workspace with azurerm_databricks_workspace. You then configure the Databricks provider against the URL that comes out of it. Only then can you lay the Unity Catalog foundation: metastore, access connector, storage credentials, external locations, catalogs, schemas and grants. Expect a Premium workspace and account admin rights before you automate that.

The same layer holds the platform-wide controls: groups, service principals, permissions, cluster policies, SQL warehouses and workspace settings. Run terraform fmt, terraform validate and terraform plan per environment, and only then terraform apply, with an approval in front of production.

databricks/
  databricks.yml
  resources/
    jobs.yml
    pipelines.yml
    dashboards.yml
  src/
    notebooks/
    python/
    sql/
  tests/
  pyproject.toml

The difference between environments sits in the targets, not in four copies of the same YAML:

bundle:
  name: customer360

include:
  - resources/*.yml

variables:
  catalog:
    description: Target catalog in Unity Catalog
  schema:
    description: Target schema

targets:
  dev:
    mode: development
    workspace:
      host: https://<dev-workspace>.azuredatabricks.net
    variables:
      catalog: dev_customer360
      schema: ${workspace.current_user.short_name}

  prod:
    mode: production
    workspace:
      host: https://<prod-workspace>.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: prod_customer360
      schema: core
    run_as:
      service_principal_name: spn-dbx-prod-deploy

The two modes genuinely do something. In development, resources get a per-developer prefix, Lakeflow pipelines are marked as development, schedules and triggers are paused and deployment locks are off so two people do not block each other. In production all of that falls away: pipelines are no longer development, compute cannot be overridden and the deployment should run under a service principal.

In your pipeline it comes down to three commands:

databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run -t dev <job_name>

Build and test your code before you deploy, and publish wheels or JARs with a version tied to your commit. That is what makes a rollback possible later: without a version number you do not know which build was running in production.

An Azure DevOps pipeline that puts the two layers in order:

trigger:
  branches:
    include:
      - main

variables:
  cliUrl: https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh

stages:
  - stage: Validate
    jobs:
      - job: ci
        steps:
          - checkout: self
          - script: |
              terraform fmt -check
              terraform validate
              databricks bundle validate -t dev
              pytest
            displayName: Validate infrastructure, bundle and code

  - stage: Terraform_Dev
    condition: succeeded()
    jobs:
      - job: tf_dev
        steps:
          - task: AzureCLI@2
            inputs:
              azureSubscription: sc-dbx-dev
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                terraform -chdir=infra/envs/dev init
                terraform -chdir=infra/envs/dev plan
                terraform -chdir=infra/envs/dev apply -auto-approve

  - stage: Bundle_Dev
    dependsOn: Terraform_Dev
    jobs:
      - job: dab_dev
        steps:
          - script: |
              curl -fsSL $(cliUrl) | sh
              databricks bundle deploy -t dev
              databricks bundle run -t dev run-unit-tests
            displayName: Install the CLI and deploy the bundle

For authentication, workload identity federation is the better choice in both worlds. In Azure DevOps it works with the AzureCLI@2 task, or through the OIDC variant with DATABRICKS_AUTH_TYPE=azure-devops-oidc alongside DATABRICKS_HOST, DATABRICKS_CLIENT_ID and SYSTEM_ACCESSTOKEN. In GitHub the Databricks CLI exchanges the workflow's OIDC token for a Databricks token. Either way there is no secret in your pipeline for somebody to rotate.

1. A developer creates a branch and changes code, tests, databricks.yml, resource YAML or a Terraform module.
2. The pull request runs linting, unit tests, terraform fmt, terraform validate, terraform plan and databricks bundle validate.
3. Merging to main starts the deployment to non-production.
4. Terraform applies the platform changes first, and only when infrastructure actually changed.
5. The bundle deploys the workload to dev or test.
6. The pipeline runs the bundle jobs for integration tests, data quality and permissions.
7. An acceptance approval guards the move to production.
8. Terraform runs production only after approval and only on infrastructure changes.
9. The bundle deploys to production in production mode under the service principal.
10. Operations watch jobs, audit logs, data quality and drift.

Rollback for the workload is a git revert followed by a new deploy, and for the platform a terraform plan and apply from the previous version. That is the argument for version numbers on your artifacts back in step 4: without that version you know what you want to roll back, but not to what.