Google BigQuery
Google BigQuery
Scan BigQuery datasets, including BigLake and Iceberg tables, without paying for full-table queries.
- Category
- Warehouse & Lakehouse
- Source type
- BIGQUERY
- Produces
- table
BigQuery is Google Cloud’s warehouse and, through BigLake, its lakehouse too. This source scans the datasets of one project: native tables, BigLake tables, BigQuery tables for Apache Iceberg, external tables over Delta or Iceberg, and views.
What you need to connect
The project id, and a service-account key in JSON, or nothing at all to use the environment’s ambient Google credentials (workload identity). The account needs BigQuery Data Viewer on the datasets, and BigQuery Job User on the project that is billed for queries.
What Classifyre reads
Every dataset and its tables and views, or only the ones you list. Each table
reports its format: native, iceberg, delta, external or view, plus its
storage location for external tables.
Shared behaviour · SQL databases
One asset per table or view, never one per row. The asset carries the table's structure — database, schema, table name, object type, its columns and their types, and a row-count estimate — and its content is a sample of real rows, formatted so a detector reads actual values rather than a schema dump.
How many rows, and which ones, is entirely up to the sampling strategy. Large tables are paged through by key rather than by OFFSET, so a scan that stops halfway can resume from where it left off instead of re-reading from the top.
Read-only throughout. The connector issues catalog queries and bounded SELECTs. Nothing is written back, and a read-only account is the right account to give it.
Relationships come out of the engine's own catalog: a view and the tables it reads from are recorded as FLOW — real lineage, with column-level detail parsed out of the view's SQL where the SQL makes that possible. See Lineage.
Metadata on every asset
Asset kind · table
| Field | Type | Always present | What it is |
|---|---|---|---|
| database | string | Yes | Database or catalog name |
| table_name | string | Yes | Table name |
| table_type | string | Yes | Object type (TABLE/VIEW) |
| schema | string | No | Schema name |
| columns | object[] | No | Columns as {name, type} objects |
| row_count | integer | No | Estimated number of rows |
| table_format | string | No | Storage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view) |
| object_type | string | No | Source object type |
| location | string | No | External data location (BigLake, external tables) |
Lineage
Lineage
Views and materialised views, and the tables they read, are recorded as lineage, parsed from the view SQL. See Lineage & Relationships.
Worth knowing
- Most reads cost nothing. ALL and AUTOMATIC page through tables with the
table-listing API rather than a query, so no bytes are billed. RANDOM and
LATEST need an
ORDER BYand run a query; set a maximum bytes billed to cap it. - A separate billing project can be named when the data and the bill live in different projects.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| project_id | string | Yes | Project whose datasets are scanned | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | Yes | —no extra properties | — |
| gcp_credentials_json | string | No | Service account JSON; empty uses application default credentials | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.billing_project_id | string | No | Project billed for queries, when it differs from project_id | — |
| connection.location | string | No | Dataset location, e.g. EU or europe-west4 | — |
| connection.maximum_bytes_billed | integer | No | Refuse any query that would bill more bytes than thismin 1 | — |
| connection.request_timeout_seconds | number | No | Timeout for one requestmin 1, max 3600 | 300 |
| extraction | object | No | —no extra properties | — |
| extraction.include_view_lineage | boolean | No | Read view definitions and record what each view selects from | true |
| scope | object | No | —no extra properties | — |
| scope.exclude_datasets | array | No | Skip these datasets | — |
| scope.exclude_datasets[] | string | No | — | — |
| scope.include_datasets | array | No | Only these datasets | — |
| scope.include_datasets[] | string | No | — | — |
| scope.include_objects | array | No | Only these objects (schema.table, or catalog.schema.table) | — |
| scope.include_objects[] | string | No | — | — |
| scope.include_tables | boolean | No | Include tables | true |
| scope.include_views | boolean | No | Include views | true |
| scope.table_limit | integer | No | Maximum number of tables to scan per runmin 1 | — |