Databricks

To register a Databricks account into TrustLogix follow the below steps

1. Create a data source entry in TrustLogix

2. Run the auto generated notebooks in Databricks environment

3. Gather all required information about Databricks account

4. Update the details in TrustLogix and Save

Create a Databricks Data Source In TrustLogix

• Login to TrustLogix console and register a new Databricks data source from Menu → Data sources → Register new data source

• Provide the data source name, & click on Register data source

• Form will be refreshed with additional fields and script Download option

Run the notebooks in Databricks environment

• Click on Download button to download the notebooks

• Extract the downloaded Zip file which will contain multiple Python notebook files

• Ensure that each of the notebook prerequisites are performed before that particular notebook is run by providing all the arguments like the Account ID, Service Principal Name, Access Token etc.

• After successful run of each of the notebooks, required objects are created in Databricks

• Capture the output of the notebook and also gather

  • Account ID

  • Workspace Deployment Name

  • SQL Warehouse Identifier

  • Service Principal Name

  • Client ID

  • Client Secret

How to register additional workspaces to an existing Databricks datasource

The following are necessary to add additional workspaces to an existing Databricks datasource

1. TrustLogix Service Principal Id

2. TrustLogix Client Id

3. TrustLogix DP Service Principal Id(Only for trustlet/hybrid tenant)

4. TrustLogix DP Client Id(Only for trustlet/hybrid tenant)

Here, 1, 2 belong to the TrustLogix Control Plane Service Principal and 3, 4 belong to the DataPlane Service Principal. You can get the above from the Results section of the initial tlx_databricks_grants notebook used while registering Databricks account in the TrustLogix console.

Step-1: Run following notebooks

1. tlx_databricks_grants_register_workspace.ipynb

2. tlx_databricks_grants_to_cp_service_principal_scheduler.ipynb(Only for trustlet/hybrid tenant)

3. tlx_databricks_workspace_admin_grant.ipynb

#1 and #3 can be run from any workspace. #2 need to be run from new workspace to be registered.

Step-2: Add the workspaces and warehouses in the TrustLogix console

• Navigate back to TrustLogix data source form page

• Update following for the new workspace

  • Workspace Deployment Name(s)

  • SQL Warehouse Identifier(s)

• Click on save, system will validate the connection details and saves them

• If all details are valid, Databricks datasource should be updated successfully

Step-3: Update secrets on AWS SSM (or Azure Key Vault) corresponding to the new workspace

tlx-<tenant-id>-<account-id>-jdbcurl
tlx-<tenant-id>-<account-id>-warehouse
tlx-<tenant-id>-<account-id>-oauthEndpoint