Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Databricks

To register a Databricks account into TrustLogix follow the below steps

1. Create a data source entry in TrustLogix

2. Run the auto generated notebooks in Databricks environment

3. Gather all required information about Databricks account

4. Update the details in TrustLogix and Save

Databricks User Permissions To Register An Account

  • Account Admin Permission: Required to register a new Databricks data source in TrustLogix. The user initiating the registration must have Account Admin privileges on the Databricks account.

  • For Registering New Workspace to an Existing Data Source

    • Workspace Admin: Required to execute the necessary notebooks for workspace registration.

    • Metastore Admin: Required to execute the necessary notebooks for workspace registration.

    Note: Both Workspace Admin and Metastore Admin permissions are required on the Databricks side to successfully register a new workspace to an existing Databricks data source.

Permissions Required by TrustLogix Service Principal(s)

  • TrustLogix uses service principals to access your Databricks environment. Two separate service principals are used depending on the feature components. The permissions required vary based on your deployment model.

  • Data Plane Service Principal Permissions for Trustlet or Hybrid tenants

    • The data plane service principal requires the following permissions on securable catalogs:

      • USE CATALOG - Allow the service principal to use the catalog

      • USE SCHEMA - Allow the service principal to use schemas within the catalog

      • BROWSE - Allow browsing of catalog objects

      • SELECT - Allow read access to tables and views

      • MANAGE - Allow management of catalog objects

      • MODIFY - Allow modification of catalog objects

  • Control Plane Service Principal Permissions for Trustlet or Hybrid tenants

    • The control plane service principal requires the following permissions on securable catalogs:

      • USE CATALOG - Allow the service principal to use the catalog

      • USE SCHEMA - Allow the service principal to use schemas within the catalog

      • BROWSE - Allow browsing of catalog objects

      • APPLY TAG - Allow applying tags to securable objects

  • SaaS Tenant Deployment

    • For SaaS tenant deployments, a single TrustLogix service principal is used with the following permissions on securable catalogs:

      • USE CATALOG - Allow the service principal to use the catalog

      • USE SCHEMA - Allow the service principal to use schemas within the catalog

      • BROWSE - Allow browsing of catalog objects

      • SELECT - Allow read access to tables and views

      • MANAGE - Allow management of catalog objects

      • MODIFY - Allow modification of catalog objects

      • APPLY TAG - Allow applying tags to securable objects

  • Important Notes on Service Principal Permissions

    • No Admin Permissions Required: TrustLogix service principals do NOT require Account Admin, Workspace Admin, or Metastore Admin permissions to function properly.

    • Principle of Least Privilege: Service principals are granted only the minimum permissions necessary to perform their designated functions.

    • Group Membership Feature Limitation: The group membership-related feature in Access Analyzer will be disabled by default for TrustLogix service principals.

    • To Enable Group Membership Feature: If you need to enable group membership-related features in Access Analyzer, the TrustLogix service principal must have at least Workspace Admin or Account Admin permissions.

Create a Databricks Data Source In TrustLogix

  • Login to TrustLogix console and register a new Databricks data source from Menu → Data sources → Register new data source

  • Provide the data source name, & click on Register data source

  • Form will be refreshed with additional fields and script Download option

Run the notebooks in Databricks environment

  • Click on Download button to download the notebooks

  • Extract the downloaded Zip file which will contain multiple Python notebook files

  • Ensure that each of the notebook prerequisites are performed before that particular notebook is run by providing all the arguments like the Account ID, Service Principal Name, Access Token etc.

  • After successful run of each of the notebooks, required objects are created in Databricks

  • Capture the output of the notebook and also gather

    • Account ID

    • Workspace Deployment Name

    • SQL Warehouse Identifier

    • Service Principal Name

    • Client ID

    • Client Secret

How to register additional workspaces to an existing Databricks data source

The following are necessary to add additional workspaces to an existing Databricks data source

  1. TrustLogix Service Principal Id

  2. TrustLogix Client Id

  3. TrustLogix DP Service Principal Id(Only for trustlet/hybrid tenant)

  4. TrustLogix DP Client Id(Only for trustlet/hybrid tenant)

  5. Workspace URL e.g adb-180424056746427.6.azuredatabricks.net

Here, 1, 2 belong to the TrustLogix Control Plane Service Principal and 3, 4 belong to the DataPlane Service Principal. You can get the above from the Results section of the initial tlx_databricks_grants notebook used while registering Databricks account in the TrustLogix console.

Step-1: Run following notebooks

  1. tlx_databricks_grants_register_workspace.ipynb

  2. tlx_databricks_grants_to_cp_service_principal_scheduler.ipynb(Only for trustlet/hybrid tenant)

  3. tlx_databricks_workspace_admin_grant.ipynb

#1 and #3 can be run from any workspace. #2 need to be run from new workspace to be registered.

To execute the above notebooks in Databricks, it is suggested that the user or service principal must have a minimum of the following permissions:

  • Workspace Admin

  • Metastore Admin

Account Admin permission not required.

Step-2: Add the workspaces and warehouses in the TrustLogix console

  • Navigate back to TrustLogix data source form page

  • Update following for the new workspace

    • Workspace Deployment Name(s)

    • SQL Warehouse Identifier(s)

  • Click on save, system will validate the connection details and saves them

  • If all details are valid, Databricks datasource should be updated successfully

Step-3: Update secrets on AWS SSM (or Azure Key Vault) corresponding to the new workspace

PreviousSnowflake with External OAuth SetupNextSQL Server

Last updated

Was this helpful?