Iceberg REST Catalog
Iceberg REST Catalog
Scan every table of an Iceberg REST catalog — Polaris, Snowflake Open Catalog, Unity Catalog, Nessie, Gravitino, Lakekeeper, S3 Tables, Glue, BigLake.
- Category
- Warehouse & Lakehouse
- Source type
- ICEBERG_REST
- Produces
- table
Most Iceberg tables are reached through a catalog rather than by path. This one source connects to any catalog that speaks the Iceberg REST protocol, lists its namespaces and tables, and reads each table as the catalog says it is now.
What you need to connect
The catalog’s REST base URL, a sign-in mode, and usually a warehouse name. If the catalog hands out credentials for its tables’ files (vended credentials), nothing else is needed; if it doesn’t, add storage keys for the bucket the tables live in.
| Catalog | Base URL | Sign-in | Warehouse |
|---|---|---|---|
| Apache Polaris | https://<host>/api/catalog | oauth2, scope PRINCIPAL_ROLE:ALL | Polaris catalog name |
| Snowflake Open Catalog | https://<account>.snowflakecomputing.com/polaris/api/catalog | oauth2, scope PRINCIPAL_ROLE:<role> | Catalog name |
| Unity Catalog | https://<workspace>/api/2.1/unity-catalog/iceberg-rest | token (personal access or service-principal token) | Unity catalog name |
| Project Nessie | https://<host>/iceberg | none or token | Branch or warehouse name |
| Apache Gravitino | https://<host>/iceberg | none or oauth2 | Optional |
| Lakekeeper | https://<host>/catalog | oauth2 or none | Warehouse name |
| AWS Glue | https://glue.<region>.amazonaws.com/iceberg | sigv4, service glue | AWS account id |
| Amazon S3 Tables | https://s3tables.<region>.amazonaws.com/iceberg | sigv4, service s3tables | Table bucket ARN |
| BigLake metastore | https://biglake.googleapis.com/iceberg/v1/restcatalog | google (service-account JSON) | gs://<bucket> |
Polaris and Gravitino have been read end to end; the others follow the same protocol but haven’t been run against the live service yet. See Lakehouse Tables for the full status.
What Classifyre reads
One asset per table, and per view unless views are switched off, across every namespace the credentials can see, or only the ones you list. Rows are read from the table’s current snapshot, so deleted rows are absent and renamed columns read as one column.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| table_format | string | No | Storage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view) |
| format_version | integer | No | Table format version (Delta reader version, Iceberg format version, Hudi table version) |
| table_version | string | No | Current version the table was read at (Delta version, Iceberg metadata file, Paimon snapshot, Hudi instant, Lance version) |
| snapshot_id | string | No | Current snapshot id (Iceberg, Paimon) |
| num_files | integer | No | Number of live data files |
| partition_columns | string[] | No | Partition column names |
| history_length | integer | No | Number of commits in the table history (Delta) |
| hudi_table_type | string | No | Hudi table type (COPY_ON_WRITE or MERGE_ON_READ) |
| namespace | string | No | Iceberg namespace |
| location | string | No | Table location in storage |
| catalog_uri | string | No | Catalog the table was found in |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Read-only. Nothing is committed to the catalog.
- Vended credentials are on by default. Turn them off when the catalog refuses to issue them, and give storage keys instead.
- OAuth2 token endpoint defaults to
<base URL>/v1/oauth/tokens; set it when your identity provider issues the tokens instead.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| auth_mode | enum | Yes | How to sign in: OAuth2 client credentials (Polaris, Snowflake Open Catalog, Lakekeeper, Gravitino), a bearer token (Unity Catalog, Nessie), AWS SigV4 (S3 Tables, Glue), Google credentials (BigLake metastore) or none (a catalog without auth) Allowed: oauth2, token, sigv4, google, none | oauth2 |
| catalog_uri | string | Yes | Base URL of the catalog's REST API, e.g. https://polaris.example.com/api/catalog, https://<workspace>/api/2.1/unity-catalog/iceberg-rest, https://glue.eu-west-1.amazonaws.com/icebergformat uri | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | Yes | Only the fields of the chosen auth_mode are used. Storage keys are needed only when the catalog does not hand out credentials for its tables' files.no extra properties | — |
| aws_access_key_id | string | No | AWS access key for SigV4 (auth_mode sigv4); empty uses the ambient AWS credentials | — |
| aws_secret_access_key | string | No | AWS secret key for SigV4 | — |
| aws_session_token | string | No | AWS session token for SigV4 | — |
| client_id | string | No | OAuth2 client id (auth_mode oauth2) | — |
| client_secret | string | No | OAuth2 client secret (auth_mode oauth2) | — |
| gcp_credentials_json | string | No | Google service account JSON (auth_mode google); empty uses application default credentials | — |
| storage_access_key_id | string | No | S3 access key for table data, when the catalog does not vend credentials | — |
| storage_azure_connection_string | string | No | Azure connection string for table data, when the catalog does not vend credentials | — |
| storage_secret_access_key | string | No | S3 secret key for table data | — |
| token | string | No | Bearer token (auth_mode token) | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.access_delegation | enum | No | Ask the catalog for credentials to read each table's files Allowed: vended-credentials, none | vended-credentials |
| connection.oauth2_scope | string | No | OAuth2 scope to request (Polaris and Open Catalog: PRINCIPAL_ROLE:ALL) | — |
| connection.oauth2_server_uri | string | No | OAuth2 token endpoint, when it is not <catalog_uri>/v1/oauth/tokensformat uri | — |
| connection.request_timeout_seconds | number | No | Timeout for one catalog requestmin 1, max 600 | 60 |
| connection.sigv4_region | string | No | AWS region for SigV4 signing | — |
| connection.sigv4_service | enum | No | AWS service name to sign for Allowed: glue, s3tables, execute-api | glue |
| connection.storage_s3_endpoint | string | No | S3 endpoint for table data when it is not AWS (MinIO, R2, ...) | — |
| connection.storage_s3_region | string | No | S3 region for table data | — |
| connection.verify_ssl | boolean | No | Verify the catalog's TLS certificate | true |
| connection.warehouse | string | No | Warehouse or catalog name the catalog expects (Polaris catalog, Lakekeeper warehouse, Unity catalog, S3 Tables bucket ARN, Glue account id) | — |
| scope | object | No | —no extra properties | — |
| scope.exclude_namespaces | array | No | Skip these namespaces | — |
| scope.exclude_namespaces[] | string | No | — | — |
| scope.include_namespaces | array | No | Only these namespaces (dot-separated for nested namespaces) | — |
| scope.include_namespaces[] | string | No | — | — |
| scope.include_objects | array | No | Only these tables (namespace.table) | — |
| scope.include_objects[] | string | No | — | — |
| scope.include_views | boolean | No | Include Iceberg views | true |
| scope.table_limit | integer | No | Maximum number of tables to scan per runmin 1 | — |