Manufacturing: Databricks and Fabric side by side with one governance layer

A manufacturer with production sites in three countries had two platforms that had grown independently of each other. The data engineering team had worked on Azure Databricks for years with Spark pipelines and machine learning, while the business analysts had started with Microsoft Fabric because reporting landed faster there. Both worked, but nobody could say with certainty where a figure in a board report came from or who had access to which table.

The trigger was an audit. The question was simple, the answer took three weeks of manual work, and that was the moment the discussion shifted from "which platform do we pick" to "how do we build one layer across both".

Databricks stays the platform for ingestion, transformation and ML. The medallion layers sit as catalogs in Unity Catalog with the three-level namespace catalog.schema.table, with one metastore per region to which that country's workspaces are attached. Processing runs on Lakeflow pipelines with serverless compute, which means the pipelines run on Unity Catalog by definition and lineage appears without anyone doing anything for it.

The Gold layer is served through a SQL warehouse. Fabric consumes that for the semantic models and Power BI reports, instead of a second copy of the same facts appearing in a second platform.

Microsoft Purview scans Unity Catalog with the Unity Catalog connector and reads lineage directly from system.access.table_lineage and system.access.column_lineage. That puts the Databricks assets in the same Data Map as the rest of the estate, and turns the audit question into a search.

Migrating was more expensive than connecting. The data engineering team had years of Spark logic that belongs on Databricks, and the analysts had found their way in Fabric. The real pain was not the existence of two platforms but the absence of one answer on lineage and access.

By taking Unity Catalog as the permission model and Purview as the overarching catalogue, the number of places where access is arranged got smaller rather than larger. Permissions at catalog level are inherited by everything below, so a new table is secured the moment it comes into existence.

Migrating everything to Fabric gave one platform and one bill, but meant rewriting Spark pipelines and ML code that demonstrably worked, plus retraining a team that was running production. The business case for that did not exist.

Keeping everything on Databricks and letting Power BI read directly was technically fine, but the analysts had moved to Fabric precisely because they could work there without going through the platform team. Reversing that solved an architecture problem and created an organisational one.

A third catalogue alongside both platforms, only for the audit, was the fastest but the worst: a list that lags behind reality from day one.

• Two platforms means two cost models side by side: capacity in Fabric and per-second consumption on Databricks. That requires two ways of steering and one joint report, otherwise each platform gets optimised without a total picture.
• The Unity Catalog connector in Purview supports incremental scans but not scoped scans. Anyone used to narrowed scans had to rethink their scan strategy.
• Serverless compute took the cluster decision out of their hands, which made costs more predictable but also less finely tunable.

Azure Databricks (Unity Catalog, Lakeflow pipelines, SQL warehouse, Delta Lake), Microsoft Purview (Data Map, lineage, classification), Microsoft Fabric (semantic models, Power BI).

• Set up Unity Catalog before you connect Purview. Purview reflects what is there and does not improve it, so a scan of a messy layout produces a messy Data Map.
• Explicitly give the scanning identity SELECT on the two lineage tables plus USE CATALOG on system and USE SCHEMA on system.access. Without those permissions the scan succeeds but lineage stays empty, and nobody notices immediately.
• Put an end date on the hive_metastore catalog. Whatever stays in it falls outside the permission model and outside lineage, and that is exactly what an audit finds.

The biggest gain was not technical but naming ownership. As soon as every catalog had an owner who could hand out permissions, the queue at the platform team disappeared and access became a decision rather than a ticket.

And a hard lesson about cost: the convenience of an all-purpose cluster for trying something out turned out to be the largest unexplainable line on the bill. Only after scheduled jobs moved to job compute and interactive compute got automatic termination did the monthly amount become predictable enough to agree on.