Using this fantastic open-source project, we have enabled GitHub Actions using ephemeral self-managed runners on AWS SPOT.

Please head over to GitHub to check out and support their excellent work.

However, one thing we have hit is docker hub limits when running multiple workflows that use containers.

We are running on AWS, so this isn't a major drama as we can pull containers to an ECS Repo and then consume them from there, but our runners need to authenticate to ECS.

To sync containers to ECS from Docker Hub, see this post.

ECS Runner Authentication

To grant our runner instances access to ECR, we need to configure the following:-

• An Instance Profile with the necessary access
• Docker Auth on Boot

Instance Profile

Create a policy similar to the following to provide the necessary access.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecr:GetRegistryPolicy",
                "ecr:DescribeImageScanFindings",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:GetDownloadUrlForLayer",
                "ecr:DescribeRegistry",
                "ecr:DescribePullThroughCacheRules",
                "ecr:DescribeImageReplicationStatus",
                "ecr:GetAuthorizationToken",
                "ecr:ListTagsForResource",
                "ecr:ListImages",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:GetRegistryScanningConfiguration",
                "ecr:BatchGetImage",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetRepositoryPolicy",
                "ecr:GetLifecyclePolicy"
            ],
            "Resource": "*"
        }
    ]
}

Docker Auth on Boot

As per the AWS documentation we need to run the following AWS CLI commands to authenticate our host to ECR enabling us to retrive containers.

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

But as we are running in AWS SPOT we need to integrate this into our OS startup scripts.

In Linux, one way to achieve this is by using rc.local. There are other ways, but this is quick and works for me.

First, we populate our file with the necessary commands.

$ cat /etc/rc.d/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
aws ecr get-login-password --region ap-southeast-2 | docker login --username AWS --password-stdin 123456789101.dkr.ecr.ap-southeast-2.amazonaws.com >> /var/log/onboot 2>&1

As per the instructions, we need to make it executable.

$ chmod 755 /etc/rc.d/rc.local

A reload and enable the service on boot, and we are done!

$ systemctl daemon-reload
$ sudo systemctl enable rc-local

I've added this to the HashiCorp Packer build spec, which bakes this into our image that will spin up on our SPOT instances whenever an Actions workflow is triggered.

When the instance starts up, we can quickly jump on via AWS Systems Manager to check our log output before the instance completes the workflow run and terminates.

$ cat /var/log/onboot
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

I'm not overly concerned with the security warning as these runners are ephemeral and will only execute a single workflow job and then terminate. The only permissions granted are to pull containers within the target AWS account.

Summary

This post shows a quick way to get authenticated access to ECR Repos from AWS SPOT instances.

It's quick and dirty but sometimes that is just what is required. It also serves as an efficient way of getting around Docker Hub limits.

I hope this helps someone else.

Cheers