Delta Lake

Required

Fields required for a valid configuration.

Path	Type	Required	Description	Default	Constraints
required	object	Yes	—	—	no extra properties
required.bucket	string	Yes	Bucket holding the tables — AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints	—	—

Masked

Sensitive fields under(secrets/credentials).

Path	Type	Required	Description	Default	Constraints
masked	object	No	Optional static credentials. Leave empty to use ambient AWS credentials chain.	—	no extra properties
masked.aws_access_key_id	string	No	S3-compatible access key ID	—	—
masked.aws_secret_access_key	string	No	S3-compatible secret access key	—	—
masked.aws_session_token	string	No	Optional session token for temporary credentials	—	—

Optional

Optional configuration fields.

Path	Type	Required	Description	Default	Constraints
optional	object	No	—	—	no extra properties
optional.connection	object	No	S3-compatible storage connection options (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, ...). Mirrors the S3 Compatible Storage source connection settings.	—	no extra properties
optional.connection.endpoint_url	string	No	Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providers	—	format uri
optional.connection.max_keys_per_page	integer	No	Maximum objects requested per provider list API call during table discovery	1000	min 1, max 1000
optional.connection.region_name	string	No	Region (recommended for AWS; required by some S3-compatible providers)	—	—
optional.connection.request_timeout_seconds	number	No	Network timeout in seconds for storage list/read operations	30	min 1, max 300
optional.connection.verify_ssl	boolean	No	TLS certificate verification toggle	true	—
optional.scope	object	No	Delta Lake table selection scope within the bucket.	—	no extra properties
optional.scope.prefix	string	No	Key prefix to search for Delta Lake tables (e.g. warehouse/). Tables are auto-discovered by their _delta_log/ directory.	—	—
optional.scope.table_limit	integer	No	Optional cap on number of table assets	—	min 1
optional.scope.table_paths	array	No	Explicit Delta Lake table root keys or s3:// URIs. When set, auto-discovery under prefix is skipped.	—	—
optional.scope.table_paths[]	string	No	—	—	—

Extracted Metadata

Table

Field	Type	Required	Description
database	string	Yes	Database or catalog name
table_name	string	Yes	Table name
table_type	string	Yes	Object type (TABLE/VIEW)
schema	string	No	Schema name
columns	object[]	No	Columns as {name, type} objects
row_count	integer	No	Estimated number of rows
format_version	integer	No	Delta protocol/reader version
num_files	integer	No	Number of data files
partition_columns	string[]	No	Partition column names
history_length	integer	No	Number of commits in the table history

Examples

Delta warehouse discovery scan

Auto-discover Delta tables under a warehouse prefix in an S3 bucket and sample rows for governance

Config Payload

{
  "type": "DELTA_LAKE",
  "required": {
    "bucket": "your-lakehouse-bucket"
  },
  "masked": {
    "aws_access_key_id": "your-access-key-id",
    "aws_secret_access_key": "your-secret-access-key"
  },
  "optional": {
    "scope": {
      "prefix": "warehouse/"
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC",
    "rows_per_page": 50
  }
}

Schedule

{
  "enabled": true,
  "preset": "nightly",
  "cron": "23 1 * * *",
  "timezone": "UTC"
}

MinIO Delta tables PII scan

Scan specific Delta tables on a MinIO (S3-compatible) endpoint for PII and secrets

Config Payload

{
  "type": "DELTA_LAKE",
  "required": {
    "bucket": "your-lakehouse-bucket"
  },
  "masked": {
    "aws_access_key_id": "your-minio-access-key",
    "aws_secret_access_key": "your-minio-secret-key"
  },
  "optional": {
    "connection": {
      "endpoint_url": "https://minio.your-company.internal:9000",
      "region_name": "us-east-1"
    },
    "scope": {
      "table_paths": [
        "warehouse/sales/orders",
        "warehouse/crm/customers"
      ]
    }
  },
  "sampling": {
    "strategy": "AUTOMATIC",
    "rows_per_page": 30
  },
  "detectors": [
    {
      "type": "PII",
      "enabled": true
    },
    {
      "type": "SECRETS",
      "enabled": true
    }
  ]
}

Schedule

{
  "enabled": true,
  "preset": "weekday_morning",
  "cron": "41 6 * * 1-5",
  "timezone": "UTC"
}