aws_s3

Process data from a S3 Bucket

RequiredFull

input:
    aws_s3:
        bucket: "my-bucket"

input:
    aws_s3:
        bucket: "my-bucket"
        prefix: "myprefix/"
        queue:
            queue_url: "https://some_queue_url"
            endpoint_url: "https://some_unique_endpoint"
            credentials:
                access_key_id: "AccessKey"
                secret_access_key: "SecretKey"
                session_token: "SessionToken"
            region: "us-west-2"
        credentials:
            access_key_id: "AccessKey"
            secret_access_key: "SecretKey"
            session_token: "SessionToken"
        delete_after_read: false
        endpoint_url: "https://some_unique_endpoint"
        read_lines: true
        force_path_style_urls: false
        region: us-east-1

Metadata

s3_bucket: S3 Bucket of the object s3_key: S3 Key

Fields

bucket

The S3 Bucket to process. Note: if queue is provided, the bucket provided in the S3 Event Record will be used
Type: string
Required: true

prefix

The prefix of the objects from the s3 bucket to process. Note: ignored if queue is used
Type: string
Required: false

queue

The aws_sqs object to utilize. Expectes S3 Notifications directly to SQS Type: object
Required: false
Properties:
    queue_url
        AWS SQS Queue URL to utilize
        Type: string
        Required: true

    endpoint_url
        Custom AWS SQS Endpoint URL
        Type: string
        Required: false

    credentials
        AWS Credentials Object to utilize. If no credentials object is provided, fiddler will utilize standard SDK locations to pull in credentials.
        Type: object
        Properties:
            access_key_id
                AWS Access Key ID
                Type: string
                Required: true

            secret_access_key
                AWS Secret Access Key
                Type: string
                Required: true

            session_token
                AWS Session Token
                Type: string
                Required: false

        Required: false

credentials

AWS Credentials Object to utilize. If no credentials object is provided, fiddler will utilize standard SDK locations to pull in credentials.
Type: object
Properties:
    access_key_id
        AWS Access Key ID
        Type: string
        Required: true

    secret_access_key
        AWS Secret Access Key
        Type: string
        Required: true

    session_token
        AWS Session Token
        Type: string
        Required: false

Required: false

delete_after_read

Delete the object after it has been fully processed
Type: bolean
Default: false Required: false

force_path_style_urls

Force Path Style S3 URLS
Type: boolean
Required: false

region

AWS Region of the bucket Type: string
Required: false

Credentials

Required IAM permissions to operate:
- s3:GetObject
- s3:ListBucket

Required IAM permissions to operate if using the Queue:
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:ChangeMessageVisibility

Required IAM permissions to operate if using delete_objects
- s3:DeleteObject

If using KMS, the following kms permissions may be needed on the relvant keys: - kms:Decrypt

retry

Retry policy for failed reads. When present, the runtime retries failed reads with backoff.

Type: object Required: false

Field	Type	Default	Description
max_retries	integer	3	Maximum retry attempts
initial_wait	string	"1s"	Wait before first retry
max_wait	string	"30s"	Maximum wait cap
backoff	string	"exponential"	Strategy: constant, linear, or exponential

With Retry

input:
  retry:
    max_retries: 3
    initial_wait: "1s"
    backoff: "exponential"
  aws_s3:
    bucket: "my-bucket"