Problem:

We want to use GitLab Container Registry in AWS ECS as private Docker images registry from which we would deploy our applications.

After creation of new ECS cluster based on EC2 instance and creation of new Task Definition with one container definition, task keep failing with message:

Status reason CannotPullContainerError: Error response from daemon: Get https://registry.gitlab.com/v2/<some_gitlab_repo>/manifests/0.0.1-SNAPSHOT: denied: access forbidden

Solution:

Create GitLab Deploy Token

First you need to create GitLab Deploy Token. To achieve this you need to:

1. Go to Settings -> CI/CD -> Deploy Tokens.
2. In 'Add a deploy token' provide name of token, when it should expire (or set it empty for 'Never') and provide username (this one would be used in AWS).
3. Check 'read_registry' checkbox and click on 'Create Deploy Token'

After creation you will get username and deploy-token. You should write it somewhere, because you will not see it again in GitLab settings.

Add deploy token to new Secrets in AWS Secret Manager

Now you need to open AWS Console, navigate to Secrets Manager (https://console.aws.amazon.com/secretsmanager/) and do below steps:

1. Click on 'Store new secret'
2. Select 'Other type of secrets'
3. Provide two variables:
   - username = <username_from_deploy_token>
   - password = <deploy_token>
4. Then click on 'Next' and on next page give name to your secrets.
5. Again click on 'Next' and then on 'Store'.

After creation of Secrets we need to copy ARN which will be useful in next steps.

Add SecretsManagerReadWrite permission to Task Definition Role

Now you need to add 'SecretsManagerReadWrite' permissions policy to Role which would be used in Task Definition. To do this you need to:

1. Navigate to Identity and Access Management (IAM) (https://console.aws.amazon.com/iam/)
2. Find or create your Role - for me it's 'ecsTaskExecutionRole'.
3. In Permissions tab click on 'Attach policies'.
4. Find 'SecretsManagerReadWrite' and click on 'Attach policy'.

Create new revision of Task Definition with ARN of Secrets to Private Registry

Now we have Role for Task Definition with correct Permissions and ARN of Secrets to GitLab Container Registry. It's time to create new revision of Task Definition.

1. Navigate to ECS Console (https://console.aws.amazon.com/ecs/).
2. Go to Task Definiton -> 'your_task_definition' -> Create new revision
3. In 'Task Role' select your Role where you added policy.
4. Then in Container Definitions click on your container and in Edit container window check 'Private repository authentication' checkbox and provide ARN to your Secrets in 'Secrets Manager ARN or name'.
5. Save everything and change in your service Task Definition Revision to this just created.

Now task should work fine and ECS sholdn't have any troubles with access to GitLab Registry.

Story:

For couple of months me and my friends are trying to work on side project. We choose to use GitLab for development, because most of us are using it daily in companies where we work. When first draft of application was done I wanted to deploy it on AWS just to learn something new (I never had opportunity to use this cloud).

After some play with creation of EC2 instance, ECS cluster, Task Definition etc. this problem occured. Because it was late at night I decided to resolve this issue next day. Unfortunately after long journey through a lot of sites / documantations etc. I didn't find easy solution for my problem.

I tried almost everything. First I configured Docker on EC2 instance and provided there deploy token. I could retrieve image from registry, but only in bash on this EC2 instance. On ECS there was still error. Then I tried to create new EC2 instance with some extra configuration injected on creation. Problem still occured. Then after two days of trying and searching Google far and wide and using couple of solutions I get that one described above.

I know that this is not a problem for someone who knows AWS, but for me it was huge challenge, because I had no clue what ARN and IAM means.

Hope this post help someone as newbie in AWS as me!