Azure Databricks: Architecture & Cost

Azure Databricks is often explained as just Spark with DBUs. That misses the point: the bill comes from architecture choices around compute, storage, networking, model serving, and governance. Whoever understands the architecture steers the cost; whoever looks only at the DBU price gets surprised.

Microsoft Learn describes Azure Databricks through a fixed layering: an account holds workspaces and Unity Catalog metastores; the control plane (managed in the Databricks account) drives the compute plane, which runs serverless in the Databricks account or classic in your own Azure subscription. Data sits in ADLS as Delta files following the medallion architecture (bronze, silver, gold). Billing runs through DBUs, metered per second; classic compute adds Azure VM, disk, and storage cost, while serverless bills serverless DBUs. On top come SQL warehouses (by size and type), Model Serving, AI Search, storage, networking (Private Link, NAT, egress), and observability. Databricks recommends serverless as the primary option and classic only where serverless does not support the workload.

The platform layers: account (identity, metastores, billing), workspace (the working environment per team or domain), Unity Catalog (governance over data and AI through the three-level namespace catalog.schema.object), control plane (web app, jobs, queries, notebooks), and compute plane (the compute power). Ingestion and pipelines run through Lakeflow (Lakeflow Connect and declarative pipelines with Auto Loader); Delta Lake provides schema enforcement, time travel, and reliable transactions. SQL warehouses come as serverless, pro, and classic, with Photon and Predictive IO. ML runs through MLflow, Model Registry, and Model Serving, with RAG and agents on top.

• Choose serverless as the default for SQL, notebooks, jobs, and Lakeflow; fall back to classic only for specific network, VM, or configuration requirements.
• Design the medallion layers (bronze, silver, gold) deliberately: batch where latency may rise, streaming where low latency or CDC is needed.
• Match SQL warehouse size to query complexity and cluster count to concurrency; making it bigger is not cost-neutral tuning.
• Co-locate data and compute in the same region and apply Private Link selectively to limit egress and data movement.

Unity Catalog is the governance anchor for data and AI: access control, ABAC, row filters and column masks, lineage, audit logs, classification, and data quality monitoring, plus AI governance through the AI Gateway. Think in seven security layers: identity and access (Entra ID, SSO, service principals, least privilege), data governance (Unity Catalog grants, workspace-catalog binding), AI governance (model endpoints, inference logging, usage limits), network security (Azure Private Link for front-end, serverless, and classic back-end), secrets and keys (customer-managed keys, encryption at rest and in transit), compute controls (cluster policies that cap max DBUs per hour and cluster settings), and compliance and monitoring (audit logs, system tables, access reviews).

The bill is architecture-driven, not only DBU-driven. Classic compute bills DBUs plus Azure VM, disk, and storage cost; serverless bills serverless DBUs. SQL warehouse cost rises sharply with size and type. On top: Model Serving (CPU or GPU DBU-hour, pay-per-token or provisioned throughput), AI Search, storage (Delta history, checkpoints, cache, logs), networking (Private Link, NAT, cross-region egress), and operational overhead (CI/CD, policies, platform team). So measure cost per pipeline run, per table refresh, per dashboard, per training run, and per inference, not only the monthly DBUs. For AI workloads, token economics counts: input and output tokens, context, model choice, retries, and orchestration can compound non-linearly, with surrounding costs such as vector databases, embeddings, and observability.

The adoption challenge is rarely whether Databricks works technically, but whether the organization has platform ownership, FinOps, governance maturity, skills, and release discipline. Microsoft's Cloud Adoption Framework stresses an adoption plan, operating model, documented responsibilities, governance, and management. Typical pilot-to-production blockers: no clearly accountable platform team, pipelines without cost attribution and release gates, ML without MLflow operationalization, security that approves networking but does not define data classification or AI governance, and finance that sees DBU spend only after the fact. Enterprise CI/CD is core, not a side topic: Terraform for the stable platform layer and Databricks Asset Bundles (now Declarative Automation Bundles) for the workload layer.

• Serverless versus classic: fast autoscaling and less infra management versus VM control and specific network or configuration requirements.
• Lakehouse versus warehouse-first: mixed data, ML, and streaming versus stable, governed SQL BI on dimensional models.
• Central versus federated governance: one metastore with standards versus domain ownership through catalogs and schemas.
• RAG versus fine-tuning: current, citable, access-filtered knowledge versus stable behavior, tone, and format.

• Seeing cost as only DBUs and forgetting VM, storage, networking, and model serving cost.
• Making a SQL warehouse bigger as tuning without looking at query complexity and concurrency.
• Editing production jobs manually, personal tokens in pipelines, and one shared catalog for all environments.
• Putting everything in Terraform instead of managing the workload layer through bundles, and skipping drift detection.
• Keeping models experimental without operationalizing MLflow, Model Registry, serving endpoints, and monitoring.