Troubleshooting AWS IAM Configuration for retrieving logs from a S3 Bucket

So, a proper IAM policy for the user you are using should be something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

  

Test your configuration

1. Make sure that you have s3cmd installed
2. Note that s3cmd will usually prefer your locally-configured s3 credentials over the ones that you provide as parameters. So, either backup your current s3 access settings, or use a new instance or Docker container.
3. Run s3cmd -- configure and enter your Logz.io IAM user access and secret keys. 
4. Try to list the bucket: 

s3cmd ls s3://<BUCKET_NAME>/<BUCKET_PREFIX>/

             (<BUCKET_PREFIX> is optional)

       5. Try to get a file from the bucket (one of those you received from the "ls" command)

s3cmd get s3://<BUCKET_NAME>/<BUCKET_PREFIX>/<OBJECT_NAME>

If both commands were successful, we should be able to pull your logs without a problem.

Important note

• Note that the ListBucket permission is set to the entire bucket and the GetObject permission ends with a /* suffix, so we can get files in subdirectories