DuckLake
DuckLake
Scan a DuckLake — its catalog in Postgres, MySQL, SQLite or DuckDB, its data in object storage.
- Category
- Warehouse & Lakehouse
- Source type
- DUCKLAKE
- Produces
- table
- Links
- DuckLakeProduct
DuckLake keeps a lakehouse’s metadata in an ordinary SQL database and its data as Parquet files in a bucket or folder. This source attaches the lake read-only and reads every table in it as it is now.
What you need to connect
The metadata catalog: a connection string such as
postgres:dbname=lake host=db.example.com user=reader, a mysql: string, a
sqlite: path, or a DuckDB file path. Its password goes in the masked password
field rather than in the string.
The data files are found where the catalog recorded them. Add storage keys for S3, Azure or GCS (HMAC) when the scanning environment can’t already read that location, and a data path when the files have moved.
What Classifyre reads
One asset per table, across every schema in the lake or only the ones you list. Rows come from the lake’s current snapshot: deleted rows are absent and a renamed column reads as one column.
Shared behaviour · SQL databases
One asset per table or view, never one per row. The asset carries the table's structure — database, schema, table name, object type, its columns and their types, and a row-count estimate — and its content is a sample of real rows, formatted so a detector reads actual values rather than a schema dump.
How many rows, and which ones, is entirely up to the sampling strategy. Large tables are paged through by key rather than by OFFSET, so a scan that stops halfway can resume from where it left off instead of re-reading from the top.
Read-only throughout. The connector issues catalog queries and bounded SELECTs. Nothing is written back, and a read-only account is the right account to give it.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| table_format | string | No | Storage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view) |
| format_version | integer | No | Table format version (Delta reader version, Iceberg format version, Hudi table version) |
| table_version | string | No | Current version the table was read at (Delta version, Iceberg metadata file, Paimon snapshot, Hudi instant, Lance version) |
| snapshot_id | string | No | Current snapshot id (Iceberg, Paimon) |
| num_files | integer | No | Number of live data files |
| partition_columns | string[] | No | Partition column names |
| history_length | integer | No | Number of commits in the table history (Delta) |
| hudi_table_type | string | No | Hudi table type (COPY_ON_WRITE or MERGE_ON_READ) |
Worth knowing
- Read-only. The lake is attached read-only; nothing is written to the catalog or the data path.
- No row estimate is shown: DuckLake’s own statistics still count rows that have since been deleted.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| catalog | string | Yes | DuckLake metadata catalog: postgres:dbname=lake host=db.example.com user=reader, mysql:..., sqlite:/path/meta.sqlite, or a DuckDB file path | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | Yes | —no extra properties | — |
| azure_connection_string | string | No | Azure connection string for the data files | — |
| catalog_password | string | No | Password for a Postgres or MySQL metadata catalog | — |
| gcs_hmac_key_id | string | No | GCS HMAC key for the data files | — |
| gcs_hmac_secret | string | No | GCS HMAC secret | — |
| s3_access_key_id | string | No | S3 access key for the data files | — |
| s3_secret_access_key | string | No | S3 secret key for the data files | — |
| s3_session_token | string | No | S3 session token | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.data_path | string | No | Where the data files are, when it differs from what the catalog recorded | — |
| connection.metadata_schema | string | No | Schema of the metadata tables in a Postgres or MySQL catalog | — |
| connection.s3_endpoint_url | string | No | S3 endpoint when it is not AWS (MinIO, R2, ...) | — |
| connection.s3_region | string | No | S3 region | — |
| scope | object | No | —no extra properties | — |
| scope.exclude_schemas | array | No | Skip these schemas | — |
| scope.exclude_schemas[] | string | No | — | — |
| scope.include_objects | array | No | Only these tables (schema.table) | — |
| scope.include_objects[] | string | No | — | — |
| scope.include_schemas | array | No | Only these schemas | — |
| scope.include_schemas[] | string | No | — | — |
| scope.table_limit | integer | No | Maximum number of tables to scan per runmin 1 | — |