Skip to Content
SourcesGoogle BigQuery

Google BigQuery

Google BigQuery

Scan BigQuery datasets, including BigLake and Iceberg tables, without paying for full-table queries.

Category
Warehouse & Lakehouse
Source type
BIGQUERY
Produces
table

BigQuery is Google Cloud’s warehouse and, through BigLake, its lakehouse too. This source scans the datasets of one project: native tables, BigLake tables, BigQuery tables for Apache Iceberg, external tables over Delta or Iceberg, and views.

What you need to connect

The project id, and a service-account key in JSON, or nothing at all to use the environment’s ambient Google credentials (workload identity). The account needs BigQuery Data Viewer on the datasets, and BigQuery Job User on the project that is billed for queries.

What Classifyre reads

Every dataset and its tables and views, or only the ones you list. Each table reports its format: native, iceberg, delta, external or view, plus its storage location for external tables.

Shared behaviour · SQL databases

One asset per table or view, never one per row. The asset carries the table's structure — database, schema, table name, object type, its columns and their types, and a row-count estimate — and its content is a sample of real rows, formatted so a detector reads actual values rather than a schema dump.

How many rows, and which ones, is entirely up to the sampling strategy. Large tables are paged through by key rather than by OFFSET, so a scan that stops halfway can resume from where it left off instead of re-reading from the top.

Read-only throughout. The connector issues catalog queries and bounded SELECTs. Nothing is written back, and a read-only account is the right account to give it.

Relationships come out of the engine's own catalog: a view and the tables it reads from are recorded as FLOW — real lineage, with column-level detail parsed out of the view's SQL where the SQL makes that possible. See Lineage.

Metadata on every asset

Asset kind · table

FieldTypeAlways presentWhat it is
databasestringYesDatabase or catalog name
table_namestringYesTable name
table_typestringYesObject type (TABLE/VIEW)
schemastringNoSchema name
columnsobject[]NoColumns as {name, type} objects
row_countintegerNoEstimated number of rows
table_formatstringNoStorage format of the table (iceberg, delta, hudi, paimon, lance, native, external, view)
object_typestringNoSource object type
locationstringNoExternal data location (BigLake, external tables)

Lineage

Lineage

Views and materialised views, and the tables they read, are recorded as lineage, parsed from the view SQL. See Lineage & Relationships.

Worth knowing

  • Most reads cost nothing. ALL and AUTOMATIC page through tables with the table-listing API rather than a query, so no bytes are billed. RANDOM and LATEST need an ORDER BY and run a query; set a maximum bytes billed to cap it.
  • A separate billing project can be named when the data and the bill live in different projects.

Configuration

Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.

Required

Without these, the source will not save.

FieldTypeRequiredWhat it doesDefault
requiredobjectYes—no extra properties—
project_idstringYesProject whose datasets are scanned—

Secrets

Stored encrypted and never shown again after you save them. See Configuration & Fields.

FieldTypeRequiredWhat it doesDefault
maskedobjectYes—no extra properties—
gcp_credentials_jsonstringNoService account JSON; empty uses application default credentials—

Optional

Everything you can tune. Sensible defaults apply when you leave them alone.

FieldTypeRequiredWhat it doesDefault
optionalobjectNo—no extra properties—
connectionobjectNo—no extra properties—
connection.billing_project_idstringNoProject billed for queries, when it differs from project_id—
connection.locationstringNoDataset location, e.g. EU or europe-west4—
connection.maximum_bytes_billedintegerNoRefuse any query that would bill more bytes than thismin 1—
connection.request_timeout_secondsnumberNoTimeout for one requestmin 1, max 3600300
extractionobjectNo—no extra properties—
extraction.include_view_lineagebooleanNoRead view definitions and record what each view selects fromtrue
scopeobjectNo—no extra properties—
scope.exclude_datasetsarrayNoSkip these datasets—
scope.exclude_datasets[]stringNo——
scope.include_datasetsarrayNoOnly these datasets—
scope.include_datasets[]stringNo——
scope.include_objectsarrayNoOnly these objects (schema.table, or catalog.schema.table)—
scope.include_objects[]stringNo——
scope.include_tablesbooleanNoInclude tablestrue
scope.include_viewsbooleanNoInclude viewstrue
scope.table_limitintegerNoMaximum number of tables to scan per runmin 1—
Last updated on