Use Databricks for synthetic data

To use Databricks as a data source or destination for your synthetic data, you need to create a Databricks connector.

Prerequisites

To create a Databricks connector, you need to obtain your SQL Warehouse connection details, a Databricks catalog name, and a personal access token for Databricks. The linked sections below provide step-by-step guidance on how to complete the prerequisites.

• Database connection details in Databricks
• Catalog name in Databricks
• Personal access token in Databricks
  📑

  Note: As MOSTLY AI leverages Unity Catalog Volumes in Databricks for data ingestion, make sure that the user associated with the access token in your destination connector is granted the CREATE TABLE and CREATE VOLUME privileges.

Get connection details for your Databricks SQL Warehouse

1. In Databricks, open the workspace that contains the SQL Warehouse you want to use.
2. Open the sidebar menu again and select SQL Warehouses.
3. From the list, open the SQL warehouse you want to use for synthetic data.
4. Select the Connection details tab.
5. Copy the necessary connection details (hostname, port, protocol, and HTTP path) for the MOSTLY AI Databricks connector.

Get Databricks catalog name

1. From the Databricks sidebar menu, select Data.
2. Copy the name of the catalog you want to use in MOSTLY AI.

Create a Databricks personal access token

1. In Databricks, open your account menu and select User Settings.

2. Under Settings, select Developer.

3. Click Manage for Access Tokens.

4. Click Generate new token.

5. In the Generate new token window, enter a name that identifies where you intend to use the token.

   💡

   Adjust the expiration of the token in the Lifetime (days) box.

6. Click Generate.

7. Copy the access token and save it in a secure location.

   ⚠️

   Before you close the window, save the token in a location you can access later.

Create a Databricks connector

c = mostly.connect({
    "name": "Databricks Connector",
    "type": "DATABRICKS",
    "access_type": "SOURCE",
    "config": {
        "host": "HOST.databricks.com",
        "http_path": "/sql/1.0/warehouses/KEY",
        "catalog": "CATALOG_NAME"
    },
    "secrets": {
        "access_token": "******"
    },
    "test_connection": True
})

Authenticate with a Service principal

To use a Service principal account to access original data stored in Databricks, you need to create a Databricks connector with a Service principal account.

config={
    "name": "Databricks connector with Service principals",
    "type": "DATABRICKS",
    "access_type": "SOURCE",
    "config": {
        "host": "INSERT_DATABRICKS_HOST",
        "http_path": "INSERT_HTTP_PATH",
        "account_name": "INSERT_ACCOUNT_NAME",
        "catalog": "INSERT_DATABRICKS_CATALOG",
        "tenant_id": "INSERT_TENANT_ID",
        "client_id": "INSERT_CLIENT_ID",
    },
    "secrets": {
        "client_secret": "INSERT_CLIENT_SECRET"
    },
    "test_connection": True
 
}
c = mostly.connect(config)

What’s next

Depending on whether you created a source or a destination connector, you can use the connector as:

• Data source for a new generator
• Data destination for a new synthetic dataset