Skip to Content
SourcesIceberg REST Catalog

Iceberg REST Catalog

Iceberg REST Catalog

Scan every table of an Iceberg REST catalog — Polaris, Snowflake Open Catalog, Unity Catalog, Nessie, Gravitino, Lakekeeper, S3 Tables, Glue, BigLake.

Category
Warehouse & Lakehouse
Source type
ICEBERG_REST
Produces
table

Most Iceberg tables are reached through a catalog rather than by path. This one source connects to any catalog that speaks the Iceberg REST protocol, lists its namespaces and tables, and reads each table as the catalog says it is now.

What you need to connect

The catalog’s REST base URL, a sign-in mode, and usually a warehouse name. If the catalog hands out credentials for its tables’ files (vended credentials), nothing else is needed; if it doesn’t, add storage keys for the bucket the tables live in.

CatalogBase URLSign-inWarehouse
Apache Polarishttps://<host>/api/catalogoauth2, scope PRINCIPAL_ROLE:ALLPolaris catalog name
Snowflake Open Cataloghttps://<account>.snowflakecomputing.com/polaris/api/catalogoauth2, scope PRINCIPAL_ROLE:<role>Catalog name
Unity Cataloghttps://<workspace>/api/2.1/unity-catalog/iceberg-resttoken (personal access or service-principal token)Unity catalog name
Project Nessiehttps://<host>/icebergnone or tokenBranch or warehouse name
Apache Gravitinohttps://<host>/icebergnone or oauth2Optional
Lakekeeperhttps://<host>/catalogoauth2 or noneWarehouse name
AWS Gluehttps://glue.<region>.amazonaws.com/icebergsigv4, service glueAWS account id
Amazon S3 Tableshttps://s3tables.<region>.amazonaws.com/icebergsigv4, service s3tablesTable bucket ARN
BigLake metastorehttps://biglake.googleapis.com/iceberg/v1/restcataloggoogle (service-account JSON)gs://<bucket>

Polaris and Gravitino have been read end to end; the others follow the same protocol but haven’t been run against the live service yet. See Lakehouse Tables for the full status.

What Classifyre reads

One asset per table, and per view unless views are switched off, across every namespace the credentials can see, or only the ones you list. Rows are read from the table’s current snapshot, so deleted rows are absent and renamed columns read as one column.

Metadata on every asset

Asset kind · table

FieldTypeAlways presentWhat it is
databasestringYesDatabase or catalog name
table_namestringYesTable name
table_typestringYesObject type (TABLE/VIEW)
schemastringNoSchema name
columnsobject[]NoColumns as {name, type} objects
row_countintegerNoEstimated number of rows
table_formatstringNoStorage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view)
format_versionintegerNoTable format version (Delta reader version, Iceberg format version, Hudi table version)
table_versionstringNoCurrent version the table was read at (Delta version, Iceberg metadata file, Paimon snapshot, Hudi instant, Lance version)
snapshot_idstringNoCurrent snapshot id (Iceberg, Paimon)
num_filesintegerNoNumber of live data files
partition_columnsstring[]NoPartition column names
history_lengthintegerNoNumber of commits in the table history (Delta)
hudi_table_typestringNoHudi table type (COPY_ON_WRITE or MERGE_ON_READ)
namespacestringNoIceberg namespace
locationstringNoTable location in storage
catalog_uristringNoCatalog the table was found in

Lineage

Lineage

This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.

Worth knowing

  • Read-only. Nothing is committed to the catalog.
  • Vended credentials are on by default. Turn them off when the catalog refuses to issue them, and give storage keys instead.
  • OAuth2 token endpoint defaults to <base URL>/v1/oauth/tokens; set it when your identity provider issues the tokens instead.

Configuration

Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.

Required

Without these, the source will not save.

FieldTypeRequiredWhat it doesDefault
requiredobjectYes—no extra properties—
auth_modeenumYesHow to sign in: OAuth2 client credentials (Polaris, Snowflake Open Catalog, Lakekeeper, Gravitino), a bearer token (Unity Catalog, Nessie), AWS SigV4 (S3 Tables, Glue), Google credentials (BigLake metastore) or none (a catalog without auth) Allowed: oauth2, token, sigv4, google, noneoauth2
catalog_uristringYesBase URL of the catalog's REST API, e.g. https://polaris.example.com/api/catalog, https://<workspace>/api/2.1/unity-catalog/iceberg-rest, https://glue.eu-west-1.amazonaws.com/icebergformat uri—

Secrets

Stored encrypted and never shown again after you save them. See Configuration & Fields.

FieldTypeRequiredWhat it doesDefault
maskedobjectYesOnly the fields of the chosen auth_mode are used. Storage keys are needed only when the catalog does not hand out credentials for its tables' files.no extra properties—
aws_access_key_idstringNoAWS access key for SigV4 (auth_mode sigv4); empty uses the ambient AWS credentials—
aws_secret_access_keystringNoAWS secret key for SigV4—
aws_session_tokenstringNoAWS session token for SigV4—
client_idstringNoOAuth2 client id (auth_mode oauth2)—
client_secretstringNoOAuth2 client secret (auth_mode oauth2)—
gcp_credentials_jsonstringNoGoogle service account JSON (auth_mode google); empty uses application default credentials—
storage_access_key_idstringNoS3 access key for table data, when the catalog does not vend credentials—
storage_azure_connection_stringstringNoAzure connection string for table data, when the catalog does not vend credentials—
storage_secret_access_keystringNoS3 secret key for table data—
tokenstringNoBearer token (auth_mode token)—

Optional

Everything you can tune. Sensible defaults apply when you leave them alone.

FieldTypeRequiredWhat it doesDefault
optionalobjectNo—no extra properties—
connectionobjectNo—no extra properties—
connection.access_delegationenumNoAsk the catalog for credentials to read each table's files Allowed: vended-credentials, nonevended-credentials
connection.oauth2_scopestringNoOAuth2 scope to request (Polaris and Open Catalog: PRINCIPAL_ROLE:ALL)—
connection.oauth2_server_uristringNoOAuth2 token endpoint, when it is not <catalog_uri>/v1/oauth/tokensformat uri—
connection.request_timeout_secondsnumberNoTimeout for one catalog requestmin 1, max 60060
connection.sigv4_regionstringNoAWS region for SigV4 signing—
connection.sigv4_serviceenumNoAWS service name to sign for Allowed: glue, s3tables, execute-apiglue
connection.storage_s3_endpointstringNoS3 endpoint for table data when it is not AWS (MinIO, R2, ...)—
connection.storage_s3_regionstringNoS3 region for table data—
connection.verify_sslbooleanNoVerify the catalog's TLS certificatetrue
connection.warehousestringNoWarehouse or catalog name the catalog expects (Polaris catalog, Lakekeeper warehouse, Unity catalog, S3 Tables bucket ARN, Glue account id)—
scopeobjectNo—no extra properties—
scope.exclude_namespacesarrayNoSkip these namespaces—
scope.exclude_namespaces[]stringNo——
scope.include_namespacesarrayNoOnly these namespaces (dot-separated for nested namespaces)—
scope.include_namespaces[]stringNo——
scope.include_objectsarrayNoOnly these tables (namespace.table)—
scope.include_objects[]stringNo——
scope.include_viewsbooleanNoInclude Iceberg viewstrue
scope.table_limitintegerNoMaximum number of tables to scan per runmin 1—
Last updated on