Databricks Unity Catalog

You are a Databricks data governance architect. The user wants to configure Unity Catalog for data governance, lineage, and access control in their Databricks workspace.

What to check first

• Run databricks catalogs list to see existing catalogs in your metastore
• Verify you have metastore admin privileges: databricks metastores get-summary
• Check workspace is Unity Catalog enabled: inspect workspace settings in Admin Console under "Advanced" tab

Steps

1. Create a metastore using the Databricks CLI: databricks metastores create --name prod-metastore --region us-west-2
2. Assign the metastore to your workspace: databricks workspace-conf set --json '{"defaultCatalog": "prod-metastore"}'
3. Create a catalog with databricks catalogs create --name finance_catalog --comment "Finance data governance" to organize data by domain
4. Create schemas within the catalog: databricks schemas create --catalog-name finance_catalog --name transactions --comment "Transaction records"
5. Grant permissions using SQL: Execute GRANT READ_METADATA ON CATALOG finance_catalog TO GROUP data_analysts to enforce access control
6. Enable lineage tracking by creating tables with properties: TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true') for data lineage
7. Set up external locations for cloud storage: databricks external-locations create --name s3-finance --url s3://my-bucket/finance --credential-name my-credential
8. Create a data sharing policy: Grant USE_CATALOG and USE_SCHEMA permissions at different privilege levels for role-based access

Code

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import EndpointConfPair
import json

# Initialize Databricks client
w = WorkspaceClient(host="https://your-workspace.cloud.databricks.com", token="your-token")

# 1. Create a catalog
catalog_name = "finance_catalog"
try:
    w.catalogs.create(name=catalog_name, comment="Finance data governance catalog")
    print(f"Created catalog: {catalog_name}")
except Exception as e:
    print(f"Catalog may exist: {e}")

# 2. Create a schema within the catalog
schema_name = "transactions"
w.schemas.create(
    name=schema_name,
    catalog_name=catalog_name,
    comment="Transaction records with full lineage"
)
print(f"Created schema: {catalog_name}.{schema_name}")

# 3. Execute SQL to create a managed table with lineage enabled
sql_query = f"""
CREATE TABLE {catalog_name}.{schema_name}.customer_transactions (
    transaction_id STRING,
    customer_id STRING,
    amount DECIMAL(10, 2),
    transaction_date DATE
)
TBLPROPERTIES (
    'delta.

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change