S3 Compatible Storage

Schema-driven source documentation.

S3_COMPATIBLE_STORAGE41 fields2 examples

Commonly Asked Questions

Assistant knowledge mapped to this source type from assistant_knowledge.json.

Required

Fields required for a valid configuration payload under `config.required`.

Path	Type	Required	Description	Default	Constraints
required	object	Yes	—	—	no extra properties
required.bucket	string	Yes	Bucket name for AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints	—	—

Masked

Sensitive fields under `config.masked` (secrets/credentials).

Path	Type	Required	Description	Default	Constraints
masked	object	No	Optional static credentials. Leave empty to use ambient AWS credentials chain.	—	no extra properties
masked.aws_access_key_id	string	No	S3-compatible access key ID	—	—
masked.aws_secret_access_key	string	No	S3-compatible secret access key	—	—
masked.aws_session_token	string	No	Optional session token for temporary credentials	—	—

Optional

Optional configuration fields under `config.optional`.

Path	Type	Required	Description	Default	Constraints
optional	object	No	—	—	no extra properties
optional.connection	object	No	—	—	no extra properties
optional.connection.endpoint_url	string	No	Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providers	—	format uri
optional.connection.max_keys_per_page	integer	No	Maximum objects requested per provider list API call	200	min 1, max 1000
optional.connection.max_object_bytes	integer	No	Maximum bytes downloaded per object for MIME detection and text extraction	5242880	min 1024, max 52428800
optional.connection.region_name	string	No	Region (recommended for AWS; required by some S3-compatible providers)	—	—
optional.connection.request_timeout_seconds	number	No	Network timeout in seconds for list/download operations	30	min 1, max 300
optional.connection.verify_ssl	boolean	No	TLS certificate verification toggle	true	—
optional.scope	object	No	Object scope and filtering controls.	—	no extra properties
optional.scope.exclude_extensions	array	No	Optional extension denylist	—	—
optional.scope.exclude_extensions[]	string	No	—	—	—
optional.scope.include_content_preview	boolean	No	Download object bytes to infer MIME and extract detector-ready text previews	true	—
optional.scope.include_empty_objects	boolean	No	Include zero-byte objects in extraction results	false	—
optional.scope.include_extensions	array	No	Optional extension allowlist (for example, .pdf, .csv, .parquet)	—	—
optional.scope.include_extensions[]	string	No	—	—	—
optional.scope.include_object_metadata	boolean	No	Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums	true	—
optional.scope.prefix	string	No	Object key prefix filter (for example, exports/2026/)	—	—

Examples

Reference payloads generated from shared source examples JSON.

S3-compatible blob scan (latest files)

Scan object metadata and file previews from an S3-compatible bucket using endpoint_url for MinIO/R2/B2

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "9 2 * * *",
  "timezone": "UTC"
}

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "customer-exports"
  },
  "masked": {
    "aws_access_key_id": "minio-access-key",
    "aws_secret_access_key": "minio-secret-key"
  },
  "optional": {
    "connection": {
      "endpoint_url": "https://storage.example.internal",
      "region_name": "us-east-1",
      "max_object_bytes": 5242880
    },
    "scope": {
      "prefix": "daily/",
      "include_extensions": [
        ".csv",
        ".parquet",
        ".pdf"
      ],
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "LATEST",
    "limit": 100
  }
}

Backblaze B2 S3 endpoint scan

Scan objects from Backblaze B2 via S3-compatible endpoint and region settings

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "14 2 * * *",
  "timezone": "UTC"
}

Config Payload

{
  "type": "S3_COMPATIBLE_STORAGE",
  "required": {
    "bucket": "b2-random-archive-bucket"
  },
  "masked": {
    "aws_access_key_id": "b2-key-id-7f3c9a4d1e",
    "aws_secret_access_key": "b2-secret-key-Qx7mN2kL9pV4sT1w"
  },
  "optional": {
    "connection": {
      "endpoint_url": "https://s3.us-west-002.backblazeb2.com",
      "region_name": "us-west-002",
      "verify_ssl": true,
      "request_timeout_seconds": 30
    },
    "scope": {
      "include_content_preview": true
    }
  },
  "sampling": {
    "strategy": "LATEST",
    "limit": 100
  }
}

Source name

Required

Authentication

Optional

Sampling