S3
  • 26 Sep 2024
  • 1 Minute to read
  • Contributors
  • Dark
    Light

S3

  • Dark
    Light

Article summary

Overview

This adapter allows you to ingest files/blobs stored in AWS S3.

Note that this adapter operates as a sink by default, meaning it will "consume" files from the S3 bucket by deleting them once ingested.

Configurations

Adapter Type: s3

  • client_options: common configuration for adapter as defined here.
  • bucket_name: the name of the bucket to ingest from.
  • access_key: an Access Key from S3 used to access the bucket.
  • secret_key: the secret key associated with the access_key used to access the bucket.
  • prefix: only ingest files with a given path prefix. Do not include a leading / in the prefix.
  • single_load: if true, the adapter will not operate as a sink, it will ingest all files in the bucket once and will then exit.

API Doc

See the official documentation.


Was this article helpful?