Government: one Databricks workspace per domain, deployed with Terraform

A large Dutch government organization with multiple business domains wanted to move to a domain-oriented data platform. Every domain had its own data analysts, its own data scientists and its own reporting requirements, but the landscape underneath had grown differently in each one. No shared standard, no automated deployment, and scaling meant someone manually clicking together another environment.

The question was not which platform to pick. The question was how domain teams could work independently without governance, security and operations being reinvented for every domain.

Every domain got its own Azure Databricks workspace. Not a shared workspace with folders per domain, but a real boundary: its own compute, its own permissions, its own deployments, and a bill that can be read per domain.

That boundary only works because something spans it. Unity Catalog sits above the workspaces as a metastore rather than inside one, so governance is shared while the working floor is separated. A table therefore has an address that does not change when another domain needs it, and access becomes a decision in the catalog instead of a copy into a second workspace.

Deployment is split in two, and that is the most important choice in this design:

• Terraform provisions the infrastructure: the workspace itself, networking, Azure Data Lake Storage Gen2, Azure Key Vault, compute policies, the metastore assignment and permissions. Reusable modules turn a new domain into a parameterized call instead of a project.
• Databricks Asset Bundles deploy the work that runs inside that workspace: notebooks, jobs and pipelines, with dev, test and production as targets in the same bundle definition.

Azure DevOps runs both, but in separate pipelines. Data engineering curates the data in Databricks into analytical datasets, and Power BI reads the Gold layer from there for reporting across the organization.

A central Cloud Platform Team manages the platform. Once a workspace exists, ownership hands over to the analysts and data scientists of that domain, who build their own data products on standards they do not have to invent themselves.

The argument in a domain-oriented platform is almost never about technology but about where the boundary sits. A shared workspace with folders per domain looks cheaper, but then a cluster that one domain runs away with is also the neighbours' cluster, and the bill cannot be attributed afterwards. A dedicated workspace makes the blast radius exactly as large as the domain and no larger.

That only holds if the boundary is not drawn by hand every time. Twenty workspaces clicked together one by one are not a platform but twenty separate environments slowly drifting apart. Terraform is therefore not a convenience here but the condition under which the choice for isolation stays sustainable.

Splitting Terraform from Asset Bundles follows the rhythm of the work. Infrastructure changes rarely and with impact, workloads change daily and with little risk. In a combined pipeline a changed notebook means a plan against the network, which makes a daily action heavy and frightening.

A single shared workspace for all domains was the fastest to set up and stayed on the table until someone asked who was allowed to change the compute policies. Every answer to that question was wrong: the platform team became a queue, or domains could reach each other's settings.

Fabric domains with OneLake shortcuts solve the same ownership question and are described elsewhere on this site. For this organization it fell away for a reason that had nothing to do with architecture: the domain teams already had their Spark and Python work on Databricks, and the gain was in standardizing that platform, not in moving it.

Creating workspaces by hand and introducing Terraform later is the route many organizations take. It works until roughly the third domain, after which nobody knows anymore which workspace deviates and why, and repairing that costs more than doing it from the start.

• A workspace per domain shrinks the blast radius but grows the number of things that must stay in step. Without shared Terraform modules the problem simply moves from manual management to maintaining twenty nearly identical definitions.
• Compute can no longer be shared across domains, so utilization per cluster drops. That costs money, and it is the price of the isolation; strict compute policies and automatic termination recover most of it.
• Domain teams get autonomy that not every team asked for. A team without engineering experience first experiences ownership as extra work, and that calls for guidance rather than more technology.
• Terraform state is now production infrastructure in its own right. Losing or polluting it creates a problem larger than the change you meant to make.

Azure Databricks (workspaces per domain, Unity Catalog, compute policies, Asset Bundles), Azure DevOps (CI/CD for infrastructure and workloads), Azure Data Lake Storage Gen2, Azure Key Vault (secrets), Microsoft Entra ID (groups and access), Power BI (reporting on the Gold layer).

• Set up Unity Catalog before the first domain gets its workspace. Fixing permissions and naming afterwards across multiple workspaces is work nobody planned for.
• Make onboarding a domain a parameterized module, not a copied folder. A copy is a deviation that has not happened yet.
• Put Terraform and Asset Bundles in separate pipelines with separate permissions. Whoever may deploy notebooks does not need to be able to change networking.
• Keep secrets in Azure Key Vault and surface them through a secret scope, so a notebook never contains a value that can end up in Git.
• Give every workspace a compute policy from day one. Adding a policy to an environment where clusters already run is a conversation; a policy that was always there is a given.

The technology was the easy part. The handover was not: provisioning a workspace takes minutes, letting ownership land takes weeks. Domain teams used to raising a ticket had to learn that they were now the ones at the controls, and that is not a technical problem.

The second lesson was about sequencing. The first two domains were delivered by hand to learn what was actually needed, which was right, but the third domain arrived before the modules were finished. That workspace deviated from the rest in three places for a year, and every change afterwards had to be thought through twice. The point where you switch from handwork to a module comes earlier than it feels.