Apache Iceberg
Apache Iceberg
Scan Apache Iceberg tables in S3-compatible storage — schema, snapshots, and row sampling, no Spark required.
- Category
- Warehouse & Lakehouse
- Source type
- ICEBERG
- Produces
- table
Iceberg is the open table format behind an increasing share of lakehouse platforms. This source reads Iceberg tables straight from the bucket they sit in, so you don’t need a REST catalog, a Glue registration, or a Spark cluster to find out what’s in them.
What you need to connect
A bucket, optional static credentials (leave them empty to use the environment’s ambient cloud credentials), and — for non-AWS S3-compatible providers — an endpoint URL and region.
Point it at a prefix to auto-discover every Iceberg table beneath it, found
by its metadata/ directory, or list exact table roots when you already know
them.
Shared behaviour · Lakehouse tables
No Spark, no JVM, and no catalog service is required. Iceberg table metadata is read directly out of the table's own files in object storage, and rows are sampled from the underlying Parquet — so the only thing you have to provide is object-storage access to the warehouse path.
Connection details are the same shape as the S3-Compatible Storage source, which means AWS S3, MinIO, Cloudflare R2 and the rest all work the same way.
What Classifyre reads
One asset per table: schema and column types, the current snapshot, partition information, and a sample of real rows.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| format_version | integer | No | Iceberg format version |
| snapshot_id | string | No | Current snapshot id |
| partition_spec | string | No | Partition spec summary |
| sort_order | string | No | Sort order summary |
| num_files | integer | No | Number of data files in the current snapshot |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Tables registered in a catalog can still be scanned this way — the source reads the table’s own metadata files, so catalog access is not required.
- Discovery is a bucket walk. Narrow the prefix, or list table paths, on buckets with millions of objects.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| bucket | string | Yes | Bucket holding the tables — AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | No | Optional static credentials. Leave empty to use ambient AWS credentials chain.no extra properties | — |
| aws_access_key_id | string | No | S3-compatible access key ID | — |
| aws_secret_access_key | string | No | S3-compatible secret access key | — |
| aws_session_token | string | No | Optional session token for temporary credentials | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | S3-compatible storage connection options (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, ...). Mirrors the S3 Compatible Storage source connection settings.no extra properties | — |
| connection.endpoint_url | string | No | Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providersformat uri | — |
| connection.max_keys_per_page | integer | No | Maximum objects requested per provider list API call during table discoverymin 1, max 1000 | 1000 |
| connection.region_name | string | No | Region (recommended for AWS; required by some S3-compatible providers) | — |
| connection.request_timeout_seconds | number | No | Network timeout in seconds for storage list/read operationsmin 1, max 300 | 30 |
| connection.verify_ssl | boolean | No | TLS certificate verification toggle | true |
| scope | object | No | Apache Iceberg table selection scope within the bucket.no extra properties | — |
| scope.prefix | string | No | Key prefix to search for Apache Iceberg tables (e.g. warehouse/). Tables are auto-discovered by their metadata/ directory. | — |
| scope.table_limit | integer | No | Optional cap on number of table assetsmin 1 | — |
| scope.table_paths | array | No | Explicit Apache Iceberg table root keys or s3:// URIs. When set, auto-discovery under prefix is skipped. | — |
| scope.table_paths[] | string | No | — | — |