S3-Compatible Storage

Required

Fields required for a valid configuration.

Path	Type	Required	Description	Default	Constraints
required	object	Yes	—	—	no extra properties
required.bucket	string	Yes	Bucket name for AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints	—	—

Masked

Sensitive fields under(secrets/credentials).

Path	Type	Required	Description	Default	Constraints
masked	object	No	Optional static credentials. Leave empty to use ambient AWS credentials chain.	—	no extra properties
masked.aws_access_key_id	string	No	S3-compatible access key ID	—	—
masked.aws_secret_access_key	string	No	S3-compatible secret access key	—	—
masked.aws_session_token	string	No	Optional session token for temporary credentials	—	—

Optional

Optional configuration fields.

Path	Type	Required	Description	Default	Constraints
optional	object	No	—	—	no extra properties
optional.connection	object	No	—	—	no extra properties
optional.connection.endpoint_url	string	No	Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providers	—	format uri
optional.connection.max_keys_per_page	integer	No	Maximum objects requested per provider list API call	200	min 1, max 1000
optional.connection.max_object_bytes	integer	No	Maximum bytes downloaded per object for MIME detection and text extraction	5242880	min 1024, max 52428800
optional.connection.region_name	string	No	Region (recommended for AWS; required by some S3-compatible providers)	—	—
optional.connection.request_timeout_seconds	number	No	Network timeout in seconds for list/download operations	30	min 1, max 300
optional.connection.verify_ssl	boolean	No	TLS certificate verification toggle	true	—
optional.scope	object	No	Object scope and filtering controls.	—	no extra properties
optional.scope.exclude_extensions	array	No	Optional extension denylist	—	—
optional.scope.exclude_extensions[]	string	No	—	—	—
optional.scope.include_content_preview	boolean	No	Download object bytes to infer MIME and extract detector-ready text previews	true	—
optional.scope.include_empty_objects	boolean	No	Include zero-byte objects in extraction results	false	—
optional.scope.include_extensions	array	No	Optional extension allowlist (for example, .pdf, .csv, .parquet)	—	—
optional.scope.include_extensions[]	string	No	—	—	—
optional.scope.include_object_metadata	boolean	No	Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums	true	—
optional.scope.prefix	string	No	Object key prefix filter (for example, exports/2026/)	—	—

Extracted Metadata

File

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
image_width	integer	No	Width in pixels
image_height	integer	No	Height in pixels
page_count	integer	No	Number of pages (pdf)
paragraph_count	integer	No	Number of paragraphs (docx)
table_count	integer	No	Number of tables (docx)
row_count	integer	No	Number of data rows
columns	object[]	No	Columns as {name, type} objects (type may be empty for csv/xlsx)
encoding	string	No	Detected character encoding
json_root_type	string	No	Root JSON type: object, array, or scalar
top_level_keys	integer	No	Number of top-level keys when the root is an object
array_length	integer	No	Length when the root is an array
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path
etag	string	No	Object entity tag

Image

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
source_hash	string	Yes	Hash of the parent object asset
location	string	Yes	Location of the embedded image within the parent

Audio

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path

Video

Field	Type	Required	Description
size_bytes	integer	Yes	Raw byte size of the content
mime_type	string	Yes	Resolved MIME type
parse_error	string	No	Set when content extraction failed
provider	string	Yes	Storage provider label
object_key	string	Yes	Object key/path

Examples

AWS S3 bucket scan

Scan an S3 bucket for document metadata and content previews using IAM credentials

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "your-bucket-name"
  },
  "masked": {
    "aws_access_key_id": "your-access-key-id",
    "aws_secret_access_key": "your-secret-access-key"
  },
  "optional": {
    "scope": {
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC"
  }
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "9 2 * * *",
  "timezone": "UTC"
}

MinIO document scan with prefix filter

Scan a MinIO bucket restricted to a key prefix and specific file types

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "your-bucket-name"
  },
  "masked": {
    "aws_access_key_id": "your-minio-access-key",
    "aws_secret_access_key": "your-minio-secret-key"
  },
  "optional": {
    "connection": {
      "endpoint_url": "https://minio.your-company.internal:9000",
      "region_name": "us-east-1"
    },
    "scope": {
      "prefix": "exports/",
      "include_extensions": [
        ".csv",
        ".parquet",
        ".pdf"
      ],
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC"
  },
  "detectors": [
    {
      "type": "PII",
      "enabled": true
    },
    {
      "type": "SECRETS",
      "enabled": true
    }
  ]
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "14 2 * * *",
  "timezone": "UTC"
}