Databricks
Databricks
Scan Unity Catalog assets, notebooks, and pipelines.
- Category
- Warehouse & Lakehouse
- Source type
- DATABRICKS
- Produces
- tablenotebookpipeline
Databricks is a lakehouse and a compute platform in one, and Unity Catalog knows more about how data moves than almost any other system Classifyre connects to — including, uniquely, which columns feed which.
What you need to connect
A workspace URL and a SQL warehouse ID (the warehouse used to sample rows), plus one of three authentication modes:
| Mode | What you supply |
|---|---|
| Personal access token | A workspace PAT |
| Service principal | Client ID and secret — the right choice for automation |
| Azure service principal | Client ID, tenant ID and secret, for Azure Databricks |
The identity needs USE CATALOG / USE SCHEMA / SELECT on what you want
scanned, and CAN USE on the SQL warehouse.
What Classifyre reads
Unity Catalog tables and views by default, scoped by catalog, schema, or an
explicit table list — including or excluding the legacy hive_metastore
catalog. Optionally also:
- Notebooks — as assets in their own right, so a credential pasted into a notebook cell is found like any other secret.
- Delta Live Tables pipelines — pipeline metadata as assets.
Shared behaviour · SQL databases
One asset per table or view, never one per row. The asset carries the table's structure — database, schema, table name, object type, its columns and their types, and a row-count estimate — and its content is a sample of real rows, formatted so a detector reads actual values rather than a schema dump.
How many rows, and which ones, is entirely up to the sampling strategy. Large tables are paged through by key rather than by OFFSET, so a scan that stops halfway can resume from where it left off instead of re-reading from the top.
Read-only throughout. The connector issues catalog queries and bounded SELECTs. Nothing is written back, and a read-only account is the right account to give it.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| catalog | string | No | Unity Catalog name |
| object_type | string | No | Source object type |
Asset kind · notebook
| Field | Type | Always present | What it is |
|---|---|---|---|
| kind | string | Yes | Asset kind discriminator (notebook) |
| path | string | Yes | Notebook workspace path |
| object_id | string | No | Workspace object id |
| language | string | No | Notebook language |
| created_at_ms | integer | No | Creation time (epoch ms) |
| modified_at_ms | integer | No | Last modified time (epoch ms) |
Asset kind · pipeline
| Field | Type | Always present | What it is |
|---|---|---|---|
| kind | string | Yes | Asset kind discriminator (pipeline) |
| pipeline_id | string | Yes | Delta Live Tables pipeline id |
| name | string | Yes | Pipeline name |
| state | string | No | Pipeline state |
Lineage
Lineage
Unity Catalog answers the lineage question directly, so nothing here is guessed from SQL text — these are among the most trustworthy lineage edges in the product.
- Table lineage records what each table was derived from.
- Column lineage is optional because it costs one request per column, but when enabled it gives genuine column-to-column mappings rather than parsed approximations.
- Upstreams in catalogs outside your scan scope are not dropped. They’re recorded by name, and if that catalog is scanned later the edge completes itself.
Worth knowing
- Sampling runs on your SQL warehouse, so scan cost is warehouse cost. A serverless warehouse with a short auto-stop is the cheapest setup.
- Column lineage on wide tables is slow. Leave it off for a first scan and turn it on once you know which catalogs matter.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
This section depends on which authentication method you pick — one of the following applies.
Personal Access Token
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "PAT_TOKEN" | Yes | — | — |
| workspace_url | string | Yes | Databricks workspace URL (for example, https://adb-1234567890123456.7.azuredatabricks.net) | — |
| warehouse_id | string | Yes | Databricks SQL warehouse ID used for sampling queries | — |
Service Principal (OAuth M2M)
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "SERVICE_PRINCIPAL" | Yes | — | — |
| workspace_url | string | Yes | Databricks workspace URL (for example, https://adb-1234567890123456.7.azuredatabricks.net) | — |
| warehouse_id | string | Yes | Databricks SQL warehouse ID used for sampling queries | — |
| client_id | string | Yes | Databricks service principal client ID | — |
Azure Service Principal
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "AZURE_SERVICE_PRINCIPAL" | Yes | — | — |
| workspace_url | string | Yes | Azure Databricks workspace URL (for example, https://adb-1234567890123456.7.azuredatabricks.net) | — |
| warehouse_id | string | Yes | Databricks SQL warehouse ID used for sampling queries | — |
| client_id | string | Yes | Azure AD application (client) ID for the service principal | — |
| tenant_id | string | Yes | Azure AD tenant ID | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
This section depends on which authentication method you pick — one of the following applies.
Databricks PAT
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| token | string | Yes | Databricks personal access token (PAT) | — |
Databricks Service Principal
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| client_secret | string | Yes | Databricks service principal client secret | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | Databricks API and SQL statement execution tuning options.no extra properties | — |
| connection.max_statement_polls | integer | No | Maximum polling attempts when waiting for SQL statement completionmin 1, max 120 | 30 |
| connection.statement_timeout_seconds | integer | No | Maximum wait timeout for SQL statement executionmin 5, max 600 | 60 |
| connection.timeout_seconds | integer | No | HTTP timeout for Databricks API callsmin 5, max 300 | 30 |
| extraction | object | No | Databricks Unity Catalog extraction feature flags.no extra properties | — |
| extraction.include_column_lineage | boolean | No | Attempt to fetch column-level lineage metadata | false |
| extraction.include_notebooks | boolean | No | Extract workspace notebook metadata as additional assets | false |
| extraction.include_pipelines | boolean | No | Extract Delta Live Tables pipeline metadata as additional assets | false |
| extraction.include_table_lineage | boolean | No | Include table-level lineage links between Unity Catalog tables | true |
| scope | object | No | Databricks Unity Catalog scope filters.no extra properties | — |
| scope.exclude_catalogs | array | No | Catalog denylist (exact catalog names) | [] |
| scope.exclude_catalogs[] | string | No | — | — |
| scope.exclude_schemas | array | No | Schema denylist. Accepted forms: schema or catalog.schema | ["information_schema"] |
| scope.exclude_schemas[] | string | No | — | — |
| scope.include_catalogs | array | No | Optional catalog allowlist (exact catalog names) | — |
| scope.include_catalogs[] | string | No | — | — |
| scope.include_hive_metastore | boolean | No | Include hive_metastore catalog in extraction | false |
| scope.include_schemas | array | No | Optional schema allowlist. Accepted forms: schema or catalog.schema | — |
| scope.include_schemas[] | string | No | — | — |
| scope.include_tables | array | No | Optional table allowlist. Accepted forms: table, schema.table, or catalog.schema.table | — |
| scope.include_tables[] | string | No | — | — |
| scope.table_limit_per_schema | integer | No | Optional cap on number of Unity Catalog tables extracted per schemamin 1 | — |