Google Cloud Storage
Google Cloud Storage
Scan objects from Google Cloud Storage buckets with ADC or service account credentials.
- Category
- Warehouse & Lakehouse
- Source type
- GOOGLE_CLOUD_STORAGE
- Produces
- fileimageaudiovideoarchive
Cloud Storage is the landing zone of the Google Cloud data stack — the bucket BigQuery exports to, the one Dataflow writes to, the one a partner drops files into.
What you need to connect
A bucket name. For credentials, either supply a service account key JSON, or supply nothing and let Application Default Credentials handle it — which on GKE means workload identity, and no secret in the configuration at all.
roles/storage.objectViewer is sufficient.
What Classifyre reads
Objects under the prefix you choose, filtered by extension where you want to narrow it.
Shared behaviour · File and object sources
Every object in the bucket becomes one asset. What it is — a PDF, a spreadsheet, a screenshot, a video — is worked out from the bytes themselves, not from the file name, because storage systems routinely label everything as generic binary data. The full list of readable types is on File Formats.
Files hidden inside other files are pulled out and scanned in their own right: images embedded in a document, and every member of a ZIP, TAR, 7z or RAR archive. Images, audio and video need OCR and transcription switched on before their content can be read.
Large objects are streamed rather than loaded whole, so a multi-gigabyte file costs disk rather than memory, and only the part a sampling window asks for is read.
Objects from this bucket are read with the shared file pipeline: see Supported File Formats for everything it can open, and OCR & Transcription for reading text out of images, audio and video.
Metadata on every asset
Asset kind · file
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| image_width | integer | No | Width in pixels |
| image_height | integer | No | Height in pixels |
| page_count | integer | No | Number of pages (pdf) |
| paragraph_count | integer | No | Number of paragraphs (docx) |
| table_count | integer | No | Number of tables (docx) |
| row_count | integer | No | Number of data rows |
| columns | object[] | No | Columns as {name, type} objects (type may be empty for csv/xlsx) |
| encoding | string | No | Detected character encoding |
| json_root_type | string | No | Root JSON type: object, array, or scalar |
| top_level_keys | integer | No | Number of top-level keys when the root is an object |
| array_length | integer | No | Length when the root is an array |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · image
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| image_width | integer | No | Width in pixels |
| image_height | integer | No | Height in pixels |
| provider | string | No | Storage provider label |
| object_key | string | No | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · audio
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · video
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · archive
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Workload identity is the recommended setup for Kubernetes deployments — nothing to store, nothing to rotate.
- A project ID override is available when the credential’s default project isn’t the one holding the bucket.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| bucket | string | Yes | Google Cloud Storage bucket name | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | No | Optional inline service account credentials JSON. Leave empty to use ADC/workload identity.no extra properties | — |
| gcp_credentials_json | string | No | Google service account credentials JSON as inline string | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.gcp_credentials_file | string | No | Path to Google service account JSON credentials file | — |
| connection.max_archive_member_bytes | integer | No | Maximum uncompressed bytes read from a single archive membermin 1024 | 10485760 |
| connection.max_archive_members | integer | No | Maximum member files expanded from one archive into child assets. Bounds fan-out, and is the guard against a zip bomb.min 1, max 10000 | 200 |
| connection.max_archive_total_bytes | integer | No | Maximum uncompressed bytes read across all members of one archive. The decompression-ratio ceiling: a zip bomb hits this before it hits memory.min 1024 | 104857600 |
| connection.max_embedded_files | integer | No | Maximum embedded files (parquet image/audio columns, office media) expanded from one container into child assets per runmin 1, max 10000 | 200 |
| connection.max_file_bytes | integer | No | Refuse any object larger than this many bytes. 0 or unset means no limit: an object above max_object_bytes is spooled to disk rather than held in memory, so file size is bounded by free disk, not by RAM.min 0 | — |
| connection.max_keys_per_page | integer | No | Maximum objects requested per list pagemin 1, max 1000 | 200 |
| connection.max_object_bytes | integer | No | Maximum bytes of one object held in memory. Larger objects are streamed to a temporary file (or read by byte range where the provider supports it), so this bounds memory rather than the size of file that can be scanned. See max_file_bytes to refuse large objects outright.min 1024 | 5242880 |
| connection.project_id | string | No | Optional GCP project ID override for auth context and bucket listing | — |
| connection.request_timeout_seconds | number | No | Network timeout in seconds for list/download operationsmin 1, max 300 | 30 |
| scope | object | No | Object scope and filtering controls.no extra properties | — |
| scope.exclude_extensions | array | No | Optional extension denylist | — |
| scope.exclude_extensions[] | string | No | — | — |
| scope.include_content_preview | boolean | No | Download object bytes to infer MIME and extract detector-ready text previews | true |
| scope.include_empty_objects | boolean | No | Include zero-byte objects in extraction results | false |
| scope.include_extensions | array | No | Optional extension allowlist (for example, .pdf, .csv, .parquet) | — |
| scope.include_extensions[] | string | No | — | — |
| scope.include_object_metadata | boolean | No | Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums | true |
| scope.prefix | string | No | Object key prefix filter (for example, exports/2026/) | — |