Delta Lake
Delta Lake
Scan Delta Lake tables in S3-compatible storage — schema, versions, and row sampling, no Spark required.
- Category
- Warehouse & Lakehouse
- Source type
- DELTA_LAKE
- Produces
- table
Delta Lake tables are the storage layer under most lakehouse platforms. This source reads them where they live — as files in a bucket — so you can scan a warehouse path without a Databricks workspace, a Spark cluster, or a catalog service in front of it.
What you need to connect
A bucket and, unless the scanning environment already has ambient cloud credentials, an access key pair. For MinIO, Cloudflare R2, Backblaze B2 and other S3-compatible providers, add the custom endpoint URL and region.
Then either give a prefix and let table discovery find every Delta table
under it (recognised by its _delta_log directory), or list exact table paths
when you know them.
Shared behaviour · Lakehouse tables
No Spark, no JVM, and no catalog service is required. Delta table metadata is read directly out of the table's own files in object storage, and rows are sampled from the underlying Parquet — so the only thing you have to provide is object-storage access to the warehouse path.
Connection details are the same shape as the S3-Compatible Storage source, which means AWS S3, MinIO, Cloudflare R2 and the rest all work the same way.
What Classifyre reads
One asset per table. Its metadata is the table’s real structure — columns and types, row count, current version — and its content is a sample of rows drawn through the sampling strategy you chose.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| format_version | integer | No | Delta protocol/reader version |
| num_files | integer | No | Number of data files |
| partition_columns | string[] | No | Partition column names |
| history_length | integer | No | Number of commits in the table history |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Reads are non-destructive and version-aware — the table’s current snapshot is read; nothing is compacted, vacuumed or written.
- Discovery costs list calls. On a very large bucket, a narrower prefix or an explicit table list is much faster than scanning from the root.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| bucket | string | Yes | Bucket holding the tables — AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | No | Optional static credentials. Leave empty to use ambient AWS credentials chain.no extra properties | — |
| aws_access_key_id | string | No | S3-compatible access key ID | — |
| aws_secret_access_key | string | No | S3-compatible secret access key | — |
| aws_session_token | string | No | Optional session token for temporary credentials | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | S3-compatible storage connection options (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, ...). Mirrors the S3 Compatible Storage source connection settings.no extra properties | — |
| connection.endpoint_url | string | No | Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providersformat uri | — |
| connection.max_keys_per_page | integer | No | Maximum objects requested per provider list API call during table discoverymin 1, max 1000 | 1000 |
| connection.region_name | string | No | Region (recommended for AWS; required by some S3-compatible providers) | — |
| connection.request_timeout_seconds | number | No | Network timeout in seconds for storage list/read operationsmin 1, max 300 | 30 |
| connection.verify_ssl | boolean | No | TLS certificate verification toggle | true |
| scope | object | No | Delta Lake table selection scope within the bucket.no extra properties | — |
| scope.prefix | string | No | Key prefix to search for Delta Lake tables (e.g. warehouse/). Tables are auto-discovered by their _delta_log/ directory. | — |
| scope.table_limit | integer | No | Optional cap on number of table assetsmin 1 | — |
| scope.table_paths | array | No | Explicit Delta Lake table root keys or s3:// URIs. When set, auto-discovery under prefix is skipped. | — |
| scope.table_paths[] | string | No | — | — |