Ctrlk
HomeLive DemoBook a DemoLogin
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Authenticate Databricks

Ensure that DataSyncs has correct permissions to access your Databricks workspace

HockeyStack connects to Databricks using OAuth machine-to-machine (M2M) authentication with a service principal. For incremental syncs (specifically for tracking new rows and changes), ensure your source table is a Delta table with Change Data Feed enabled.

What you need

Before configuring the Data Sync, prepare:

  • A Databricks service principal assigned to the target workspace.

  • An OAuth secret generated for that service principal.

  • A Databricks SQL warehouse that the service principal can use.

  • One exact Unity Catalog table that the service principal can read.

An account or workspace administrator normally performs this setup.

1. Create Service Principal and Generate OAuth Credentials

A workspace administrator must set up the service principal. As a workspace admin, log in to the Databricks workspace and:

  • Create or select a service principal.

  • Assign it to the workspace containing the SQL warehouse and source table.

In the service principal details page:

  • Click the Secrets tab

  • Under OAuth secrets, click Generate secret

    • Set the secret's lifetime in days (maximum 730 days)

  • Important: Copy both values immediately as Databricks displays them only once:

    • Client ID: (the service principal's application ID)

    • Client secret: (the generated OAuth secret)

    • Click Done when finished

Databricks displays the secret only once. Store it securely and do not add it to source control, documentation, logs, or screenshots.

2. Grant Required Permissions

Grant the service principal:

  • CAN USE on the selected SQL warehouse.

  • USE CATALOG on the table's catalog.

  • USE SCHEMA on the table's schema.

  • SELECT on the source table.

These privileges can be granted directly, inherited from a parent object, or provided through a group. Follow least privilege: the service principal only needs access to the warehouse and source data used by the import.

3. Find Your SQL Warehouse Connection Details

In your Databricks workspace:

  • Go to SQL > SQL Warehouses

  • Select the warehouse you created for DataSyncs

  • Click Connection Details

  • Copy the Server hostname (e.g., dbc-a1b2c3d4-e5f6.cloud.databricks.com)

  • Copy the HTTP path (e.g., /sql/1.0/warehouses/a1b2c3d4e5f67890)

Enter the hostname in HockeyStack as a full HTTPS workspace URL, for example:

The HTTP path must identify a SQL warehouse, for example:

HockeyStack supports commercial Azure, AWS, and GCP Databricks workspace URLs. Use the workspace URL, not an account-console URL, and do not append /api or another path.

Fields to Enter in HockeyStack

Collect the following information to enter in HockeyStack:

Field
Value
Example

Workspace URL

Full HTTPS workspace URL

https://dbc-a1b2c3d4-e5f6.cloud.databricks.com

SQL Warehouse HTTP path

HTTP path from warehouse Connection Details

/sql/1.0/warehouses/a1b2c3d4e5f67890

Catalog

Exact Unity Catalog name

prod

Schema

Exact schema name in that catalog

analytics

Table

Exact source table name

customer_data

OAuth Client ID

Service principal application ID

(from step 2)

OAuth Client secret

OAuth secret from step 2

(from step 2)

Important: Catalog, schema, and table are separate fields. Enter their exact names—do not enter a combined value like catalog.schema.table or use wildcards.

Test Your Connection

After completing all fields in HockeyStack's Data Sync configuration:

  1. Click Test Connection.

  2. HockeyStack verifies:

    • The OAuth client ID and secret are valid

    • Access to the selected SQL warehouse

    • Access to the exact catalog, schema, and table

    • The source table schema for use with mappings and filters

Changing a connection, credential, or table field after a successful test requires another connection test.

IP Whitelisting (optional)

If your Databricks workspace has IP restrictions, whitelist the following HockeyStack IPs:

  • 18.184.228.143

  • 18.192.106.69

  • 35.157.54.242

Common Failures

  • OAuth credentials rejected: verify the client ID, recopy the secret, check that it has not expired, and ensure the service principal is assigned to the workspace. Do not add spaces around the client ID or secret.

  • Warehouse unavailable or forbidden: confirm the workspace URL and HTTP path and grant CAN USE on the warehouse.

  • Table cannot be accessed: verify the exact catalog, schema, and table names and grant USE CATALOG, USE SCHEMA, and SELECT.

  • Timeout or connection error: verify that the workspace and warehouse are available and that network controls allow HTTPS access to Databricks.

For incremental new rows imports, the source must also be a Delta table with Change Data Feed enabled. HockeyStack does not enable it automatically.

Security

  • HockeyStack treats the client secret as a password and never displays a saved secret.

  • Saved connection details do not expose the client ID, client secret, OAuth tokens, or internal import offsets.

  • OAuth M2M (machine-to-machine) is the only supported authentication method for this integration.

Databricks References

PreviousAuthenticate Snowflake
NextAuthenticate Redshift - Export Only

Last updated