Databricks

Required

Fields required for a valid configuration.

Path	Type	Required	Description	Default	Constraints
required	object	Yes	—	—	—

Masked

Sensitive fields under(secrets/credentials).

Path	Type	Required	Description	Default	Constraints
masked	object	Yes	—	—	—

Optional

Optional configuration fields.

Path	Type	Required	Description	Default	Constraints
optional	object	No	—	—	no extra properties
optional.connection	object	No	Databricks API and SQL statement execution tuning options.	—	no extra properties
optional.connection.max_statement_polls	integer	No	Maximum polling attempts when waiting for SQL statement completion	30	min 1, max 120
optional.connection.statement_timeout_seconds	integer	No	Maximum wait timeout for SQL statement execution	60	min 5, max 600
optional.connection.timeout_seconds	integer	No	HTTP timeout for Databricks API calls	30	min 5, max 300
optional.extraction	object	No	Databricks Unity Catalog extraction feature flags.	—	no extra properties
optional.extraction.include_column_lineage	boolean	No	Attempt to fetch column-level lineage metadata	false	—
optional.extraction.include_notebooks	boolean	No	Extract workspace notebook metadata as additional assets	false	—
optional.extraction.include_pipelines	boolean	No	Extract Delta Live Tables pipeline metadata as additional assets	false	—
optional.extraction.include_table_lineage	boolean	No	Include table-level lineage links between Unity Catalog tables	true	—
optional.scope	object	No	Databricks Unity Catalog scope filters.	—	no extra properties
optional.scope.exclude_catalogs	array	No	Catalog denylist (exact catalog names)	[]	—
optional.scope.exclude_catalogs[]	string	No	—	—	—
optional.scope.exclude_schemas	array	No	Schema denylist. Accepted forms: schema or catalog.schema	["information_schema"]	—
optional.scope.exclude_schemas[]	string	No	—	—	—
optional.scope.include_catalogs	array	No	Optional catalog allowlist (exact catalog names)	—	—
optional.scope.include_catalogs[]	string	No	—	—	—
optional.scope.include_hive_metastore	boolean	No	Include hive_metastore catalog in extraction	false	—
optional.scope.include_schemas	array	No	Optional schema allowlist. Accepted forms: schema or catalog.schema	—	—
optional.scope.include_schemas[]	string	No	—	—	—
optional.scope.include_tables	array	No	Optional table allowlist. Accepted forms: table, schema.table, or catalog.schema.table	—	—
optional.scope.include_tables[]	string	No	—	—	—
optional.scope.table_limit_per_schema	integer	No	Optional cap on number of Unity Catalog tables extracted per schema	—	min 1

Extracted Metadata

Table

Field	Type	Required	Description
database	string	Yes	Database or catalog name
table_name	string	Yes	Table name
table_type	string	Yes	Object type (TABLE/VIEW)
schema	string	No	Schema name
columns	object[]	No	Columns as {name, type} objects
row_count	integer	No	Estimated number of rows
catalog	string	No	Unity Catalog name
object_type	string	No	Source object type

Notebook

Field	Type	Required	Description
kind	string	Yes	Asset kind discriminator (notebook)
path	string	Yes	Notebook workspace path
object_id	string	No	Workspace object id
language	string	No	Notebook language
created_at_ms	integer	No	Creation time (epoch ms)
modified_at_ms	integer	No	Last modified time (epoch ms)

Pipeline

Field	Type	Required	Description
kind	string	Yes	Asset kind discriminator (pipeline)
pipeline_id	string	Yes	Delta Live Tables pipeline id
name	string	Yes	Pipeline name
state	string	No	Pipeline state

Examples

Unity Catalog scan with PAT token

Discover all catalogs, schemas, and tables in Databricks Unity Catalog with table lineage enabled

Config Payload

{
  "type": "DATABRICKS",
  "required": {
    "auth_mode": "PAT_TOKEN",
    "workspace_url": "https://your-workspace.azuredatabricks.net",
    "warehouse_id": "your-warehouse-id"
  },
  "masked": {
    "token": "your-pat-token"
  },
  "optional": {
    "extraction": {
      "include_table_lineage": true,
      "include_column_lineage": false,
      "include_notebooks": false,
      "include_pipelines": false
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC",
    "rows_per_page": 50
  }
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "26 0 * * *",
  "timezone": "UTC"
}

Service principal with full metadata extraction

Production scan using service principal auth with lineage, notebooks, and pipeline metadata

Config Payload

{
  "type": "DATABRICKS",
  "required": {
    "auth_mode": "SERVICE_PRINCIPAL",
    "workspace_url": "https://your-workspace.azuredatabricks.net",
    "warehouse_id": "your-warehouse-id",
    "client_id": "your-service-principal-client-id"
  },
  "masked": {
    "client_secret": "your-service-principal-client-secret"
  },
  "optional": {
    "extraction": {
      "include_table_lineage": true,
      "include_column_lineage": true,
      "include_notebooks": true,
      "include_pipelines": true
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC",
    "fallback_to_random": true,
    "rows_per_page": 50
  }
}

Schedule

{
  "enabled": true,
  "preset": "weekday_business",
  "cron": "8 15 * * 1-5",
  "timezone": "UTC"
}

Azure service principal with tenant-wide scan

Scan Azure Databricks workspace using Azure AD service principal authentication with catalog scope filtering

Config Payload

{
  "type": "DATABRICKS",
  "required": {
    "auth_mode": "AZURE_SERVICE_PRINCIPAL",
    "workspace_url": "https://adb-1234567890123456.7.azuredatabricks.net",
    "warehouse_id": "your-warehouse-id",
    "client_id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    "tenant_id": "ffffffff-gggg-hhhh-iiii-jjjjjjjjjjjj"
  },
  "masked": {
    "client_secret": "your-azure-client-secret"
  },
  "optional": {
    "scope": {
      "include_catalogs": [
        "production",
        "analytics"
      ],
      "exclude_schemas": [
        "information_schema"
      ],
      "table_limit_per_schema": 100
    },
    "extraction": {
      "include_table_lineage": true,
      "include_column_lineage": false,
      "include_notebooks": true,
      "include_pipelines": false
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC",
    "rows_per_page": 50
  },
  "detectors": [
    {
      "type": "PII",
      "enabled": true
    }
  ]
}

Schedule

{
  "enabled": true,
  "preset": "weekday_business",
  "cron": "42 14 * * 1-5",
  "timezone": "UTC"
}