Skip to Content
SourcesLakehouse Tables

Lakehouse Tables

A lakehouse table is a folder of data files plus the metadata that says which of those files, and which rows in them, make up the table right now. Reading the data files one by one gives the wrong answer: deleted rows are still in them, older snapshots keep their own files, and a renamed column keeps its old name in everything written before the rename.

Classifyre always reads the table, not its files. Each table becomes one asset with its real columns, its current row count where the format records it, and a sample of its current rows.

Which source to use

Where the table isUse
Files in S3, MinIO, R2, B2 or another S3-compatible bucketS3-Compatible Storage
Files in Azure Blob Storage, ADLS Gen2 or Fabric OneLakeAzure Blob Storage
Files in Google Cloud StorageGoogle Cloud Storage
Files on a mounted diskLocal Folder
An Iceberg REST catalog (Polaris, Unity, Glue, S3 Tables, Nessie, …)Iceberg REST Catalog
A DuckLakeDuckLake
Behind Trino or StarburstTrino
Behind Athena and the Glue Data CatalogAmazon Athena
BigLake or BigQuery tables for Apache IcebergGoogle BigQuery
Redshift Spectrum external tablesAmazon Redshift
Databricks Unity Catalog, Snowflake, Hive, DremioTheir own sources: Databricks, Snowflake, Hive, Dremio

A bucket scan is the simplest route and needs no catalog at all. A catalog or engine source is the right one when access is granted through the catalog, when tables are spread across many buckets, or when you want the names people use for the tables rather than their paths.

Every table asset, whichever source found it, carries a table_format in its metadata: delta, iceberg, hudi, paimon, lance, ducklake, native for the engine’s own tables, view, or external for plain files behind an external table.

Formats in a bucket

What a bucket or folder scan reads, by storage. Verified means it was read end to end against real or emulated storage during development; supported means the code path exists but hasn’t been run against that service yet.

FormatS3-compatibleAzure / OneLakeGCSLocal folder
Delta Lake (deletion vectors, column mapping)VerifiedVerified (Azurite); OneLake supportedSupported, needs an HMAC keyVerified
Apache Iceberg v1/v2 (merge-on-read deletes)VerifiedVerified (Azurite)Supported, needs an HMAC keyVerified
LanceVerifiedVerified (Azurite)SupportedVerified
Apache PaimonVerifiedNot readable yetSupportedVerified
Apache Hudi (copy-on-write snapshot)SupportedSupportedSupportedSupported

Known limits:

  • Paimon on Azure isn’t readable yet; such tables are logged as skipped.
  • Delta and Iceberg on GCS are read through GCS’s S3-compatible interface, which needs an HMAC key pair on the source. Without one they are logged as skipped.
  • Row counts are reported only where the table’s own metadata states them exactly: Delta (deletion vectors subtracted), Iceberg snapshots with no delete files, Paimon append tables and Lance. Otherwise the count is left empty rather than guessed.
  • An Iceberg table found by path is read through its newest metadata.json. A catalog may already have moved on to a newer one it keeps elsewhere (Gravitino’s in-memory store does this); use the Iceberg REST Catalog source for those.

Catalogs

The Iceberg REST Catalog source works with any catalog that speaks the Iceberg REST protocol.

CatalogSign-inStatus
Apache PolarisOAuth2 client credentialsVerified
Apache GravitinoNone or OAuth2Verified
Snowflake Open CatalogOAuth2 client credentialsSupported
Databricks Unity Catalog (Iceberg endpoint)Bearer tokenSupported
Project NessieNone or bearer tokenSupported
LakekeeperOAuth2 or noneSupported
AWS Glue Iceberg endpointAWS SigV4 (glue)Supported
Amazon S3 TablesAWS SigV4 (s3tables)Supported
Google BigLake metastoreGoogle service accountSupported

Engines

EngineWhat it reportsStatus
Trino / StarburstEvery catalog: Iceberg, Delta, Hudi, Hive and the restVerified (Trino)
Amazon AthenaGlue tables: Iceberg, Delta, Hudi, plain filesSupported
Google BigQueryNative, BigLake Iceberg, external Delta and IcebergSupported
Amazon RedshiftLocal tables and Spectrum external tablesSupported
DatabricksDelta and Iceberg in Unity CatalogSupported
SnowflakeNative and Iceberg tablesSupported
HiveHive, Iceberg, Delta and Hudi tables in the metastoreSupported
DremioIts datasets and their formatsSupported

Next steps

  • Reading the rows of these tables, and joining them with your databases — SQL
  • What a bucket scan reads besides tables — File Formats
  • How many rows are read per run — Sampling Strategies
Last updated on