Skip to main content
|
Best PracticeLevel: Advanced

Unity Catalog governance best practices

What keeps governance on Databricks sustainable in production: permission inheritance, managed tables, system tables and saying goodbye to the Hive metastore.

TechExplained 8 min readPublished: 15 August 2026Last updated: 15 August 2026
#unity catalog#governance#security#lineage#databricks
Session on Unity Catalog governance in a meeting room: the steps from inheritance and ownership through managed tables to system tables on screen, a hand-drawn catalog model on the flip chart
01

Let permissions inherit instead of handing them out

Unity Catalog works with a hierarchy: the metastore at the top, and below it the three-level namespace catalog.schema.table. Permissions you grant high up carry through to everything below. That is not a detail but the entire design: if you arrange access at catalog level, a new table is automatically secured correctly the moment it comes into existence.

The anti-pattern is granting per table because it feels fastest at that moment. Six months later nobody can explain why somebody has access to something, and cleaning up is riskier than leaving it alone.

02

Audit your existing MANAGE grants

Since 3 August 2026 MANAGE needs fewer usage privileges. MANAGE on a catalog now works without USE CATALOG on that same catalog, and MANAGE on a schema still asks for USE CATALOG on the parent catalog but no longer for USE SCHEMA on that schema. On a table, view, volume or function, USE CATALOG and USE SCHEMA on the parents are still required.

That sounds like a relaxation and it is one, but it has a tail. Grants that used to do nothing because the matching usage privilege was missing are now active. Anyone who once handed out MANAGE generously on the assumption that it had no effect without USE CATALOG has gained administrators since, without anybody changing a thing. So walk through your MANAGE grants once and confirm they are intended.

The relaxation applies only to the metadata capabilities of MANAGE and of child privileges such as READ METADATA. Data access does not change: SELECT and MODIFY still require USE CATALOG and USE SCHEMA, even for someone holding MANAGE. That distinction is exactly what makes READ METADATA useful for auditors: read access to grants, row filters and masks, without ownership and without sight of the data itself.

03

Grant write access as small as the workload allows

Since 24 August 2026 write access no longer has to be a synonym for MODIFY. Unity Catalog has INSERT, UPDATE and DELETE as separate privileges in Beta. They grant access to a table's data and not to its schema or other metadata, which lets you record for the first time the difference between a pipeline that appends rows and a process that may empty the table.

Use them where the workload only does one thing, usually a service principal behind an ingest pipeline. Do account for the operations that require two privileges: INSERT OVERWRITE, INSERT REPLACE WHERE and dynamic partition overwrite need INSERT and DELETE together, and TRUNCATE falls under DELETE. Anything touching the schema still requires MODIFY, including schema evolution.

The most important agreement is not about granting but about revoking. MODIFY is a composite privilege and these three are its children, and they are managed independently. REVOKE MODIFY therefore does not remove a separately granted DELETE. Have your access review look at all four rather than only at MODIFY, otherwise it returns a clean result on a table where someone can still delete rows. See Revoking MODIFY does not remove a separately granted DELETE.

04

Grant access to groups and name an owner

Permissions granted to people are permissions that stay in place when that person moves team. Work with security groups, and record who owns each catalog. The owner is the one who can hand out permissions, so without a named owner every request lands on the platform team.

05

Choose deliberately between accumulating and replacing

The model above accumulates. Somebody in three groups can reach everything those three groups can reach, and that accumulated set travels with them. For most organisations that is exactly right: it is predictable and it scales.

There is one situation where it does not work, and that is separation that must not mix. Think of somebody working for two clients, or a researcher on two studies. Until recently the only real answer was a second workspace or even a second metastore, with duplicate setup as the price. Since 19 August 2026 role-based access control is generally available: a user assumes a role (technically a group they hold Assume on), and that role's permissions replace their own set for the duration of the session.

Do not switch it on if you do not need it, because it costs work and it has edges. The most important one: every group in a workspace inherits from the users system group, so by default a role still sees everything shared with all workspace users. Anyone promising exclusive access trims that system group first. Also check whether your workloads can cope, because Lakeflow pipelines do not work under a role. See What Azure Databricks RBAC cannot do yet.

06

Put skills in a catalog of their own

Since 28 August 2026 an agent skill is a securable in the three-level namespace, catalog.schema.skill, in Beta. That adds an object type you have to place, and it goes wrong faster than it does with a table.

Skills did not get privileges of their own. Their contents sit in Unity Catalog managed storage, so you govern them with the volume privileges CREATE VOLUME, WRITE VOLUME and READ VOLUME, on top of the usual USE CATALOG and USE SCHEMA. Two consequences belong in your design. First, WRITE VOLUME alone is not enough to update a skill: always grant READ VOLUME alongside it. Second, and this weighs more, CREATE VOLUME is one privilege. Grant it on a domain schema so someone may publish skills, and that same person may also create ordinary volumes there.

So create a separate catalog for skills, with a schema per team or domain. That is what Databricks suggests as well, and it is the only way to assign authorship without handing out privileges on your data plane along with it. Before you let anyone publish, check that the schema sits under a standard catalog, so not under a foreign or federated one, and that it resolves to a managed storage location; without that location creation fails. Finally, bear in mind that workspaces under the compliance security profile may read skills but not create, update or delete them.

07

Make managed tables the default

Databricks recommends managed tables for most new tables, because the platform then manages the lifecycle, the file layout and the optimization. You get maintenance you would otherwise have to schedule yourself.

External tables remain necessary in three situations: data that already sits somewhere, data reached by path from tools outside Databricks, and formats managed tables do not support. Foreign tables are read-only references through Lakehouse Federation. Choose deliberately, because the type determines who does the maintenance.

08

Put an end date on the Hive metastore

The legacy Hive metastore appears in the namespace as the hive_metastore catalog. Everything in it falls outside your Unity Catalog permission model and outside your lineage. Because it works, it stays.

Treat this as debt with a repayment schedule: an end date and a counter on the number of tables still left. Workspaces created after 9 November 2023 were automatically enabled for Unity Catalog by Databricks, older ones were not, so verify rather than assume.

09

Use the system tables as evidence

Lineage sits in system.access.table_lineage and system.access.column_lineage. That is not only where Purview reads it from, it is also where you can read it from. When somebody asks which reports lean on a table before you change it, that is a query and not an estimate.

Arrange read access explicitly for the account that scans or reports: SELECT on both tables, plus USE CATALOG on system and USE SCHEMA on system.access.

Since 26 August 2026, do not count on the literal SQL being included. Azure Databricks masks statement_text in system.query.history as <REDACTED> by default, does the same to query_text in the Query History API, and omits the keys in request_params in system.access.audit that contain a SQL definition. Only account admins and members of the account group databricks_pii_access see the text, and you have to create that group yourself.

That hits your evidence less hard than it sounds, provided you know where the boundary sits. The other columns and the other keys in request_params are unchanged, so who accessed what and when is still a query. What now requires a membership is what exactly was executed. So walk through your existing dashboards, alerts and jobs that lean on statement_text, because they do not fail but quietly deliver empty text. See Reading query text from system tables requires membership of a group that does not exist.

10

Connect Purview once the layout is settled and not before

Purview reflects what is in Unity Catalog and does not improve it. A scan of a messy layout produces a messy Data Map. Set up your catalogs first, then do the connection. See How to connect Azure Databricks to Microsoft Purview.

The process at a glance

Click a step for its key decision

Summary

Let permissions inherit instead of handing them out

Unity Catalog works with a hierarchy: the metastore at the top, and below it the three-level namespace catalog.schema.table. Permissions you grant high up carry through to everything below. That is not a detail but the entire design: if you arrange access at catalog level, a new table is automatically secured correctly the moment it comes into existence.

Production Readiness Checklist

Before you go live, verify these production recommendations.

  • One metastore per region, with workspaces attached to it
  • Permissions at catalog or schema level, not per table
  • Access through security groups rather than per person
  • An owner named for every catalog
  • MANAGE grants audited after the August 2026 relaxation
  • Write access as small as the workload allows, and the revoke verified against it
  • A deliberate choice between accumulating permissions and roles that replace them
  • Managed tables as the default, external with a reason
  • Skills in a catalog of their own, not in a schema holding data
  • hive_metastore has an end date and a counter
  • Lineage verified through the system tables
  • Reporting on statement_text verified since the August 2026 masking
Unity Catalog governance best practices | TechExplained