🔗 How IdP Groups Are Tied to Databricks Groups (Unity Catalog)

🔑 Key Principle (Read This First)

Databricks does NOT “map” IdP groups to Databricks groups manually.
The linkage happens through SCIM provisioning.

SCIM = the binding glue between IdP and Databricks.

---

1️⃣ High-Level Flow

Identity Provider (Okta / Azure AD)
        |
        |  SCIM Provisioning
        v
Databricks Account Console
        |
        |  Group sync
        v
Unity Catalog Authorization

• IdP creates & owns the group

• SCIM syncs it into Databricks

• Unity Catalog grants privileges to the group

• Databricks enforces access

---

2️⃣ Where Each Thing Is Defined

Item	Where It Lives
Groups (finance-readers)	IdP (Okta / Azure AD)
Group membership	IdP
Group sync	SCIM
Group visibility	Databricks Account Console
Data privileges	Unity Catalog (SQL)

---

3️⃣ Step-by-Step: Tie IdP Groups to Databricks

---

STEP 1: Create Groups in the IdP

Example: Azure AD / Okta

Create these groups:

• finance-readers

• finance-writers

• finance-engineers

• finance-data-owners

Add users and service principals only in the IdP.

📌 Databricks should never be the source of truth.

---

STEP 2: Enable SCIM Provisioning in Databricks

In Databricks Account Console:

1. Go to User Management

2. Enable SCIM provisioning

3. Generate SCIM Token

4. Copy SCIM endpoint URL

📌 This is one-time setup.

---

STEP 3: Configure SCIM in the IdP

Example: Azure AD

• Add Databricks SCIM app

• Configure:

  • SCIM endpoint

  • Bearer token

• Assign:

  • Groups

  • Users

  • Service principals

Example: Okta

• Enable SCIM provisioning

• Assign groups to the Databricks app

• Push groups & memberships

✔ Groups now auto-sync.

---

STEP 4: Verify Groups in Databricks

Since you’re not an admin, ask an admin to verify in:

Databricks Account Console
→ User Management
→ Groups

Or verify yourself using SQL:

SHOW GROUPS;

You should now see:

finance-readers
finance-writers
finance-engineers

These groups are:
✔ SCIM-managed
✔ Read-only in Databricks
✔ Governed by IdP

---

STEP 5: Grant Unity Catalog Privileges to SCIM Groups

Now comes the binding to data.

GRANT USE CATALOG ON CATALOG finance TO `finance-readers`;
GRANT USE SCHEMA ON SCHEMA finance.gl TO `finance-readers`;

GRANT SELECT ON TABLE finance.gl.transactions TO `finance-readers`;
GRANT SELECT, MODIFY ON TABLE finance.gl.transactions TO `finance-writers`;

GRANT CREATE TABLE ON SCHEMA finance.gl TO `finance-engineers`;

🎯 This is where “roles” become real.

---

4️⃣ How Membership Changes Are Enforced (Important)

Change	Where Done	Result
User added to group	IdP	Access granted automatically
User removed	IdP	Access revoked automatically
User terminated	IdP	Immediate loss of access
New user onboarded	IdP	Group membership applies

🚀 No Databricks admin action required.

---

5️⃣ Service Principals (ETL / Genie)

Same exact model.

In IdP:

• Create service account / app registration

• Add to group finance-etl-sp

SCIM:

• Syncs service principal

Databricks:

GRANT MODIFY ON TABLE finance.gl.transactions TO `finance-etl-sp`;

Jobs and Genie now run securely.

---

6️⃣ How to Tell If a Group Is SCIM-Managed

In SQL:

DESCRIBE GROUP `finance-readers`;

You’ll see:

• External ID

• Read-only membership

📌 If it’s editable → it’s a local group (anti-pattern).

---

7️⃣ Common Mistakes (Avoid These 🚫)

❌ Manually creating groups in Databricks for prod
❌ Adding users directly in Databricks
❌ Granting privileges to individual users
❌ Using workspace-local groups
❌ Mixing SCIM and local groups

---

8️⃣ One-Screen Mental Model

IdP (truth) → SCIM → Databricks Groups → Unity Catalog Grants → Enforcement

---