Data & AI · October 2026
Databricks Unity Catalog for Regulated Enterprises: Governance, Access Control & Compliance
15 min read
Most enterprises that deploy Databricks on Azure get the compute right and the governance wrong. They build medallion architectures, optimize Spark jobs, and ship production pipelines — then fail a SOC 2 audit because no one can answer basic questions: who has access to the customer PII table, when did that access change, and what downstream models consumed that data last quarter? Unity Catalog is the answer to all three. But only if it's designed correctly from the start.
Unity Catalog is Databricks' unified governance layer — a single control plane for data access, lineage, auditing, and discovery across all your Databricks workspaces. For regulated enterprises, it's not optional. It's the difference between a data platform that passes audit and one that generates findings. This guide covers the design decisions that matter: metastore architecture, access control model, lineage configuration, Microsoft Purview integration, and the compliance controls that auditors actually check.
What Unity Catalog Actually Is
Before the design decisions, the mental model: Unity Catalog introduces a three-level namespace — catalog, schema, table — that sits above the workspace level. A single Unity Catalog metastore can govern multiple Databricks workspaces across multiple regions. Every securable object (table, view, volume, function, model) lives in this hierarchy and inherits permissions from it.
This matters because it solves the problem that plagued Databricks before Unity Catalog: workspace-level Hive metastores. In the old model, every workspace had its own metastore, its own access control, and its own lineage — or more accurately, no lineage. A data engineer in the analytics workspace had no visibility into what the data science workspace was doing with the same underlying tables. Permissions were managed per-workspace with no central audit trail. Unity Catalog collapses all of that into a single governance plane.
For regulated enterprises, the implication is significant: one place to define who can access what data, one audit log for all data access across all workspaces, and one lineage graph that traces data from source to model to dashboard.
Metastore Architecture: One Per Region
The first design decision is metastore topology. Unity Catalog metastores are regional — one metastore per Azure region. All workspaces in that region attach to the same metastore and share its governance model. For most enterprises, this means one metastore per region where you operate Databricks.
The mistake we see: creating multiple metastores in the same region to separate environments (dev, staging, prod). This defeats the purpose. Unity Catalog is designed to govern across environments using catalog-level isolation — a dev catalog, a staging catalog, and a prod catalog within the same metastore. Permissions, lineage, and audit logs flow across all three. You get a complete picture of how data moves from development through production, which is exactly what BCBS 239 and SOC 2 CC6.1 require.
The metastore root storage account is a critical infrastructure decision. Unity Catalog stores managed table data and system tables in an Azure Data Lake Storage Gen2 account that you own. This account must be in the same region as the metastore, must use hierarchical namespace, and must be accessible via a managed identity — not a service principal with a secret. In regulated environments, this storage account needs the same controls as the rest of your data estate: private endpoint, encryption with customer-managed keys, diagnostic logging, and immutable storage for audit logs.
Catalog Design: Isolation Without Fragmentation
Catalogs are the top-level namespace in Unity Catalog and the primary isolation boundary. The design question is how many catalogs to create and what they represent. Two patterns work well for regulated enterprises:
Environment-based catalogs. One catalog per environment: dev, staging, prod. Data engineers work in dev, promotion pipelines move data through staging, production is locked down with strict access controls. Simple, predictable, easy to audit. Works well for organizations with a single data domain or a small number of teams.
Domain-based catalogs. One catalog per business domain: finance, risk, operations, shared. Each domain owns its catalog and manages access within it. Cross-domain access is explicit — the risk team grants the finance team SELECT on specific tables, and that grant is audited. Works well for large enterprises with data mesh architectures or strict data ownership requirements.
For most regulated enterprises, we recommend a hybrid: domain-based catalogs with environment suffixes (finance_prod, finance_dev). It gives you domain ownership and environment isolation without the complexity of a full data mesh governance model.
Access Control: The Model That Scales
Unity Catalog uses a privilege model with inheritance — permissions granted at the catalog level flow down to schemas and tables unless overridden. This is powerful but requires intentional design. The wrong approach is granting permissions directly to individual users. The right approach is group-based access control mapped to Entra ID.
Entra ID group mapping
Unity Catalog syncs users and groups from Entra ID via SCIM provisioning. Every Databricks user is an Entra ID user. Every Databricks group is an Entra ID group. Permissions in Unity Catalog are granted to groups, not individuals. When someone joins the data engineering team, they get added to the Entra ID group — and immediately inherit the correct Unity Catalog permissions. When they leave, their Entra ID account is disabled and all access is revoked automatically.
This is the access control model that satisfies SOC 2 CC6.2 and CC6.3 — provisioning and deprovisioning tied to a single identity system, with no orphaned accounts and no manual permission cleanup. Your auditor will ask for evidence of this. Unity Catalog's system tables provide it automatically.
Row-level and column-level security
Unity Catalog supports row filters and column masks — the controls that regulated industries need for PII and sensitive data. A row filter restricts which rows a user can see based on their group membership. A column mask replaces sensitive values with masked output for users who don't have clearance to see the raw data.
In practice: your customer table has a ssn column. Data analysts can query the table but see XXX-XX-1234. Data engineers with PII clearance see the full value. The masking logic is defined once in Unity Catalog and applied consistently across every query, every workspace, every tool that connects to Databricks. No application-level masking that can be bypassed. No separate security views to maintain.
For HIPAA-covered entities, this is the technical safeguard that satisfies the minimum necessary standard — users see only the data they need for their role, enforced at the platform level.
Dynamic views vs. row filters
Before Unity Catalog, the standard approach for row-level security in Databricks was dynamic views — views that embedded current_user() logic to filter rows. This works but creates maintenance overhead: every table that needs row-level security needs a corresponding view, and those views need to be kept in sync with the underlying table schema.
Row filters in Unity Catalog replace dynamic views for most use cases. The filter is attached to the table itself, not a view on top of it. Users query the table directly and the filter is applied transparently. Simpler to maintain, easier to audit, and the lineage graph shows the actual table — not a view that obscures the data source.
Keep dynamic views for complex access patterns that row filters can't express. Use row filters for everything else.
Data Lineage: The Compliance Artifact
Unity Catalog captures column-level lineage automatically — no instrumentation required. Every query that runs on a Unity Catalog table is tracked: which columns were read, which columns were written, which notebook or job executed the query, and which user ran it. This lineage graph is queryable via system tables and visualizable in the Databricks UI.
For regulated enterprises, lineage is not a nice-to-have. BCBS 239 requires financial institutions to demonstrate data lineage for risk data — where did this number come from, what transformations were applied, and who approved the logic? SOC 2 CC6.1 requires evidence that data flows are documented and controlled. Unity Catalog's automatic lineage satisfies both without manual documentation.
The practical implication: when your risk model produces a capital requirement number, you can trace that number back through every dbt transformation, every source table, every upstream pipeline — automatically. When an auditor asks “how was this number calculated?” the answer is a lineage graph, not a Word document that may or may not be current.
Lineage across tools
Unity Catalog lineage extends beyond Databricks notebooks and jobs. dbt models running on Databricks SQL warehouses contribute lineage automatically — dbt's column-level lineage integrates with Unity Catalog's lineage graph. Power BI reports connected to Databricks SQL endpoints appear in the lineage graph as downstream consumers. The result is an end-to-end lineage picture: raw source data → bronze table → silver transformation → gold model → Power BI dashboard.
This cross-tool lineage is what makes Unity Catalog genuinely useful for compliance — not just tracking what happens inside Databricks, but tracking the full data journey from ingestion to consumption.
Microsoft Purview Integration
Unity Catalog and Microsoft Purview serve complementary roles. Unity Catalog governs access and tracks lineage within the Databricks platform. Purview provides enterprise-wide data discovery, classification, and governance across your entire Azure estate — Databricks, Azure SQL, Azure Data Factory, Azure Synapse, and every other data source.
The integration works via the Purview connector for Databricks: Purview scans Unity Catalog metadata, imports table schemas and lineage, and applies sensitivity labels based on Purview's classification rules. A table tagged as containing PII in Purview surfaces that classification in Unity Catalog — and Unity Catalog can enforce access policies based on Purview sensitivity labels.
For regulated enterprises, this integration closes the governance gap between your data platform and your broader Azure security posture. Purview becomes the single catalog for all enterprise data. Unity Catalog becomes the enforcement layer for Databricks-specific access. They share metadata and classifications without duplicating governance logic.
Configure the Purview scan on a schedule — daily for production catalogs, weekly for dev. Set up automated classification rules for common PII patterns (SSN, credit card numbers, email addresses, dates of birth). Enable sensitivity label propagation so that when Purview classifies a column as Highly Confidential, Unity Catalog automatically applies the corresponding column mask.
Audit Logging: What Auditors Actually Check
Unity Catalog writes all governance events to system tables — access grants, permission changes, data access events, and lineage updates. These system tables are queryable via SQL and can be forwarded to your SIEM via Databricks audit log delivery to Azure Storage or Event Hubs.
The audit events that matter most for regulated industries:
- Permission changes: Who granted or revoked access to what, and when. Required for SOC 2 CC6.2 and most financial services regulatory frameworks
- Data access events: Every SELECT on a sensitive table, with user identity, timestamp, and query text. Required for HIPAA audit controls and PCI DSS requirement 10
- Schema changes: DDL events — table creation, column additions, schema modifications. Required for change management evidence under SOC 2 CC8.1
- Failed access attempts: Queries that were denied due to insufficient permissions. These are the events your SOC team needs to investigate for insider threat detection
Configure audit log delivery to Azure Storage with immutable storage policies — logs cannot be modified or deleted once written. Retention period should match your regulatory requirement: 7 years for BSA/AML, 6 years for HIPAA, 1 year minimum for SOC 2. Forward to Microsoft Sentinel for real-time alerting on anomalous access patterns.
The Migration: From Hive Metastore to Unity Catalog
If you have existing Databricks workspaces running on the legacy Hive metastore, migration to Unity Catalog is not optional — Databricks has deprecated the Hive metastore for new features, and Unity Catalog is the only path to column-level lineage, row filters, and centralized governance.
The migration path: enable Unity Catalog on the metastore, upgrade workspaces to Unity Catalog mode, then migrate tables using the Unity Catalog migration tool. The tool handles table metadata migration and can sync external table locations. What it doesn't handle automatically: access control migration (Hive metastore permissions don't map cleanly to Unity Catalog privileges), and any code that uses legacy Hive-specific syntax.
Plan the migration in phases. Start with new tables — all new tables go into Unity Catalog from day one. Then migrate gold-layer tables (the ones downstream consumers query). Then silver. Bronze last. This sequence minimizes disruption because gold tables have the most stable schemas and the most predictable access patterns.
The access control migration is the hardest part. Hive metastore permissions were workspace-scoped and often granted to individual users. Unity Catalog permissions should be group-scoped and Entra ID-backed. Use the migration as an opportunity to rationalize access — don't just replicate the old permissions, design the right model and migrate to it.
What Good Looks Like
A well-governed Databricks platform on Azure looks like this: a single Unity Catalog metastore per region, with domain-based catalogs and environment isolation. All users and groups provisioned from Entra ID via SCIM. Permissions granted to groups, never individuals. Row filters and column masks on every table containing PII or sensitive financial data. Automatic lineage captured for every query, queryable by auditors on demand. Purview scanning Unity Catalog daily and propagating sensitivity labels. Audit logs delivered to immutable Azure Storage and forwarded to Sentinel.
When your next SOC 2 audit comes around, the evidence package for data access controls is a SQL query against Unity Catalog system tables — not a manual spreadsheet compiled the week before the audit. That's the difference Unity Catalog makes for regulated enterprises.
If you're running Databricks on Azure without Unity Catalog, or you have Unity Catalog deployed but not configured for compliance, the time to address it is before your auditors find the gap — not after.
Running Databricks in a regulated environment?
Talk to an architect who has designed Unity Catalog governance models for financial services, healthcare, and government organizations.
Schedule a Discovery Call