S3

Overview

This adapter allows you to ingest files/blobs stored in AWS S3.

Note that this adapter operates as a sink by default, meaning it will "consume" files from the S3 bucket by deleting them once ingested.

Configurations

Adapter Type: s3

• client_options: common configuration for adapter as defined here.
• bucket_name: the name of the bucket to ingest from.
• access_key: an Access Key from S3 used to access the bucket.
• secret_key: the secret key associated with the access_key used to access the bucket.
• prefix: only ingest files with a given path prefix. Do not include a leading / in the prefix.
• single_load: if true, the adapter will not operate as a sink, it will ingest all files in the bucket once and will then exit.

API Doc

See the official documentation.