S3 Compatible Storage
Schema-driven source documentation.
S3_COMPATIBLE_STORAGE48 fields2 examples
Commonly Asked Questions
Assistant knowledge mapped to this source type from
assistant_knowledge.json.Required
Fields required for a valid configuration payload under `config.required`.
| Path | Type | Required | Description | Default | Constraints |
|---|---|---|---|---|---|
| required | object | Yes | — | — | no extra properties |
| required.bucket | string | Yes | Bucket name for AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints | — | — |
Masked
Sensitive fields under `config.masked` (secrets/credentials).
| Path | Type | Required | Description | Default | Constraints |
|---|---|---|---|---|---|
| masked | object | No | Optional static credentials. Leave empty to use ambient AWS credentials chain. | — | no extra properties |
| masked.aws_access_key_id | string | No | S3-compatible access key ID | — | — |
| masked.aws_secret_access_key | string | No | S3-compatible secret access key | — | — |
| masked.aws_session_token | string | No | Optional session token for temporary credentials | — | — |
Optional
Optional configuration fields under `config.optional`.
| Path | Type | Required | Description | Default | Constraints |
|---|---|---|---|---|---|
| optional | object | No | — | — | no extra properties |
| optional.connection | object | No | — | — | no extra properties |
| optional.connection.endpoint_url | string | No | Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providers | — | format uri |
| optional.connection.max_keys_per_page | integer | No | Maximum objects requested per provider list API call | 200 | min 1, max 1000 |
| optional.connection.max_object_bytes | integer | No | Maximum bytes downloaded per object for MIME detection and text extraction | 5242880 | min 1024, max 52428800 |
| optional.connection.region_name | string | No | Region (recommended for AWS; required by some S3-compatible providers) | — | — |
| optional.connection.request_timeout_seconds | number | No | Network timeout in seconds for list/download operations | 30 | min 1, max 300 |
| optional.connection.verify_ssl | boolean | No | TLS certificate verification toggle | true | — |
| optional.scope | object | No | Object scope and filtering controls. | — | no extra properties |
| optional.scope.exclude_extensions | array | No | Optional extension denylist | — | — |
| optional.scope.exclude_extensions[] | string | No | — | — | — |
| optional.scope.include_content_preview | boolean | No | Download object bytes to infer MIME and extract detector-ready text previews | true | — |
| optional.scope.include_empty_objects | boolean | No | Include zero-byte objects in extraction results | false | — |
| optional.scope.include_extensions | array | No | Optional extension allowlist (for example, .pdf, .csv, .parquet) | — | — |
| optional.scope.include_extensions[] | string | No | — | — | — |
| optional.scope.include_object_metadata | boolean | No | Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums | true | — |
| optional.scope.prefix | string | No | Object key prefix filter (for example, exports/2026/) | — | — |
Examples
Reference payloads generated from shared source examples JSON.
AWS S3 bucket scan
Scan an S3 bucket for document metadata and content previews using IAM credentials
Schedule
{
"enabled": true,
"preset": "nightly",
"cron": "9 2 * * *",
"timezone": "UTC"
}Config Payload
{
"type": "S3_COMPATIBLE_STORAGE",
"required": {
"bucket": "your-bucket-name"
},
"masked": {
"aws_access_key_id": "your-access-key-id",
"aws_secret_access_key": "your-secret-access-key"
},
"optional": {
"scope": {
"include_content_preview": true
}
},
"sampling": {
"strategy": "LATEST"
}
}MinIO document scan with prefix filter
Scan a MinIO bucket restricted to a key prefix and specific file types
Schedule
{
"enabled": true,
"preset": "nightly",
"cron": "14 2 * * *",
"timezone": "UTC"
}Config Payload
{
"type": "S3_COMPATIBLE_STORAGE",
"required": {
"bucket": "your-bucket-name"
},
"masked": {
"aws_access_key_id": "your-minio-access-key",
"aws_secret_access_key": "your-minio-secret-key"
},
"optional": {
"connection": {
"endpoint_url": "https://minio.your-company.internal:9000",
"region_name": "us-east-1"
},
"scope": {
"prefix": "exports/",
"include_extensions": [
".csv",
".parquet",
".pdf"
],
"include_content_preview": true
}
},
"sampling": {
"strategy": "LATEST"
},
"detectors": [
{
"type": "PII",
"enabled": true
},
{
"type": "SECRETS",
"enabled": true
}
]
}