Amazon Athena
Amazon Athena
Scan the AWS Glue Data Catalog through Amazon Athena — Iceberg, Delta, Hudi and file tables in S3.
- Category
- Warehouse & Lakehouse
- Source type
- ATHENA
- Produces
- table
On AWS, the Glue Data Catalog is the list of what’s in the lake, and Athena is the way to read it. This source lists databases and tables from Glue and reads rows with Athena SQL, so each format is read by its own reader.
What you need to connect
The region and an S3 location for query results (Athena writes every query’s result there). Give an access key pair, or leave it empty to use the environment’s ambient AWS credentials. The identity needs Glue read access, Athena query access, read access to the tables’ data, and write access to the results location.
What Classifyre reads
Every Glue database and its tables and views, or only the ones you list. Each
table reports its format from Glue: iceberg, delta, hudi, external for
plain files, or view, plus its storage location.
Shared behaviour · SQL databases
One asset per table or view, never one per row. The asset carries the table's structure — database, schema, table name, object type, its columns and their types, and a row-count estimate — and its content is a sample of real rows, formatted so a detector reads actual values rather than a schema dump.
How many rows, and which ones, is entirely up to the sampling strategy. Large tables are paged through by key rather than by OFFSET, so a scan that stops halfway can resume from where it left off instead of re-reading from the top.
Read-only throughout. The connector issues catalog queries and bounded SELECTs. Nothing is written back, and a read-only account is the right account to give it.
Relationships come out of the engine's own catalog: a view and the tables it reads from are recorded as FLOW — real lineage, with column-level detail parsed out of the view's SQL where the SQL makes that possible. See Lineage.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| table_format | string | No | Storage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view) |
| object_type | string | No | Source object type |
| location | string | No | Table location in S3 |
Lineage
Lineage
Views and the tables they read are recorded as lineage, from the SQL Athena stores with each view. See Lineage & Relationships.
Worth knowing
- Athena bills by bytes scanned, and a row limit doesn’t reduce what a scan of an unpartitioned table reads. Every page of rows is one query, so the page size is also the cost knob, and RANDOM sampling reads the whole table once per run.
- Another data catalog registered in Athena can be named in place of the
default
AwsDataCatalog.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| region | string | Yes | AWS region, e.g. eu-west-1 | — |
| s3_staging_dir | string | Yes | S3 location Athena writes query results to, e.g. s3://athena-results-bucket/classifyre/ | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | Yes | Leave empty to use the scanning environment's AWS credentials (instance role, IRSA).no extra properties | — |
| aws_access_key_id | string | No | AWS access key; empty uses the ambient AWS credentials | — |
| aws_secret_access_key | string | No | AWS secret key | — |
| aws_session_token | string | No | AWS session token | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.catalog_name | string | No | Data catalog to scan | AwsDataCatalog |
| connection.request_timeout_seconds | number | No | Timeout for one querymin 1, max 3600 | 300 |
| connection.work_group | string | No | Athena workgroup | primary |
| extraction | object | No | —no extra properties | — |
| extraction.include_view_lineage | boolean | No | Read view definitions and record what each view selects from | true |
| scope | object | No | —no extra properties | — |
| scope.exclude_databases | array | No | Skip these Glue databases | — |
| scope.exclude_databases[] | string | No | — | — |
| scope.include_databases | array | No | Only these Glue databases | — |
| scope.include_databases[] | string | No | — | — |
| scope.include_objects | array | No | Only these objects (schema.table, or catalog.schema.table) | — |
| scope.include_objects[] | string | No | — | — |
| scope.include_tables | boolean | No | Include tables | true |
| scope.include_views | boolean | No | Include views | true |
| scope.table_limit | integer | No | Maximum number of tables to scan per runmin 1 | — |