AWS CloudTrail has a wealth of information that often gets forgotten and unchecked.

An experienced Cloud Engineer will regularly check the trail for any issues brewing that might get them woken up at 3 am!

Victor Grenu posted the following tweet some months back, and ever since, it's just a copy/paste away when I'm troubleshooting any issue.

This time I stumbled across an issue with an AWS generated IAM Policy within CloudWatch Synthetic Canaries.

The Issue

We run an application with a Front-End that is both internally and externally visible, which makes it a perfect use to monitor the availability of each entry point using the Canary Heartbeat Monitoring Blueprint.

One of the nice things about canaries is that you can provision them within your VPC and on the internet, as they are just Lambdas under the covers.

Once configured, you'll get an excellent status dashboard but can also set up alarms and get screenshots of runs that fail etc.

When you configure the Canary, it sets up an IAM Role and S3 bucket to store various configs and the output from each run.

Anyway, back to our issue.

Log Insights Query

Let's run our query and see if anything looks untoward. Oh dear, that doesn't look very healthy.

We can see that our auto-generated IAM role seems to be having issues with the GetBucketLocation call.

Here we find the issue.

The first statement of the IAM policy allows the s3:GetBucketLocation action; however, the resource is bound to an object path within the bucket.

This will cause the API query to fail, as the resource must explicitly be the S3 bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::cw-syn-results-1234567891011-region/canary/canary-prod-ext/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "arn:aws:logs:region:1234567891011:log-group:/aws/lambda/cwsyn-canary-prod-ext-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": "cloudwatch:PutMetricData",
            "Condition": {
                "StringEquals": {
                    "cloudwatch:namespace": "CloudWatchSynthetics"
                }
            }
        }
    ]
}

## The Fix

This is easy to fix by adding the s3:GetBucketLocation to the following policy section below. However, I will log this out with AWS so they can fix it, so it doesn't keep happening.

```json
{
    "Effect": "Allow",
    "Action": [
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
    ],
    "Resource": [
        "*"
    ]
}

Once applied our errors stop and CloudTrail Xen returns.

Summary

This post shows how powerful taking a peek at your CloudTrail is; hopefully, you will now put it on the list to do more regularly!

I hope this helps someone else.

Cheers