Create a AWS ECS Cluster with AWS CLI Tools

In this tutorial we will create a AWS ECS Cluster using the AWS CLI Tools.

Create ECS Cluster

I assume that you have a AWS IAM User with Full Access to EC2, ECS and VPC and that your AWS CLI tools has been configured.

If not, you can consult the following documentation:

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam.html
• https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

We will create the ECS Cluster called tools-cluster :

$ $ aws --profile default ecs create-cluster --cluster-name tools-cluster
{
    "cluster": {
        "clusterArn": "arn:aws:ecs:eu-west-1:012345678901:cluster/tools-cluster",
        "clusterName": "tools-cluster",
        "status": "ACTIVE",
        "registeredContainerInstancesCount": 0,
        "runningTasksCount": 0,
        "pendingTasksCount": 0,
        "activeServicesCount": 0,
        "statistics": [],
        "tags": [],
        "settings": [
            {
                "name": "containerInsights",
                "value": "disabled"
            }
        ],
        "capacityProviders": [],
        "defaultCapacityProviderStrategy": []
    }
}

Get the Subnet IDs from your VPC, in my case my subnets are tagged with private in their naming convention:

$ aws --profile default ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-01234567" | jq -r '.Subnets[] | select(.Tags[].Value | contains("private")) .SubnetId'
subnet-0123456789aaaaaaa
subnet-0123456789bbbbbbb
subnet-0123456789ccccccc

Now we will create the ECS Instance Role Policy:

$ cat ecs_instance_role_policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Then create the IAM Role named ecs-tools-instance-role:

$ aws --profile default iam create-role --role-name ecs-tools-instance-role --assume-role-policy-document file://ecs_instance_role_policy.json

{
    "Role": {
        "Path": "/",
        "RoleName": "ecs-tools-instance-role",
        "RoleId": "XX",
        "Arn": "arn:aws:iam::xxxxxxxxxxxx:role/ecs-tools-instance-role",
        "CreateDate": "2020-05-19T10:24:18+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ec2.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Attach the AmazonEC2ContainerServiceforEC2Role to the Role ecs-tools-instance-role:

$ aws --profile default iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role --role-name ecs-tools-instance-role

Create the Instance Profile Role:

$ aws --profile default iam create-instance-profile --instance-profile-name ecs-tools-instance-role
{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "ecs-tools-instance-role",
        "InstanceProfileId": "XX",
        "Arn": "arn:aws:iam::xxxxxxxxxxxx:instance-profile/ecs-tools-instance-role",
        "CreateDate": "2020-05-19T10:30:36+00:00",
        "Roles": []
    }
}

Add the IAM Role to Instance Profile:

$ aws --profile default iam add-role-to-instance-profile --instance-profile-name ecs-tools-instance-role --role-name ecs-tools-instance-role

Create the ECS Task Role Policy ecs_tasks_role_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the Task IAM Role:

$ aws --profile default iam create-role --role-name ecs-tools-task-role --assume-role-policy-document file://ecs_tasks_role_policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "ecs-tools-task-role",
        "RoleId": "X",
        "Arn": "arn:aws:iam::xxxxxxxxxxxx:role/ecs-tools-task-role",
        "CreateDate": "2020-05-19T15:43:58+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Attach any policies to the IAM Role that you prefer, in my case SSMReadOnly Policy:

$ aws --profile default iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchFullAccess  --role-name ecs-tools-task-role

$ aws --profile default iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy  --role-name ecs-tools-task-role

$ aws --profile default iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess --role-name ecs-tools-task-role

Get the Instance Profile ARN:

$ aws --profile default iam list-instance-profiles-for-role --role-name ecs-tools-instance-role --query 'InstanceProfiles[0].Arn' | jq -r ''
arn:aws:iam::xxxxxxxxxxxx:instance-profile/ecs-tools-instance-role

Create your first Task Definition, taskdef.json:

{
  "family": "mytaskdefinition",
  "containerDefinitions": [{
    "name": "myapp",
    "image": "httpd:2.4",
    "cpu": 68,
    "portMappings": [{
      "containerPort": 80,
      "hostPort": 80,
      "protocol": "tcp"
    }],
    "memory": 50,
    "essential": true,
    "entryPoint": [
      "sh",
      "-c"
    ],
    "command": [
      "/bin/sh -c \"echo 'hello from ecs container instance' >  /usr/local/apache2/htdocs/index.html && httpd-foreground\""
    ]
  }]
}

Register your ECS Task Definition:

$ aws --profile default ecs register-task-definition --cli-input-json file://taskdef.json

Create ECS Container Instance

Get the latest ECS optimized AMI:

$ aws --profile default ssm get-parameter --name '/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id' | jq -r '.Parameter.Value'
ami-0a74b180a0c97ecd1

Create a SSH Keypair (optional):

$ aws --profile default ec2 create-key-pair --key-name ecs --query 'KeyMaterial' --output text > ecs.pem
$ chmod 400 ecs.pem

Create the userdata for your ECS Container Instance:

$ cat userdata.txt
#!/bin/bash
echo "ECS_CLUSTER=tools-cluster" >> /etc/ecs/ecs.config
echo ECS_AVAILABLE_LOGGING_DRIVERS='["json-file","awslogs"]' >> /etc/ecs/ecs.config
echo "ECS_CONTAINER_INSTANCE_TAGS={\"ECS_CLUSTER\": \"tools-cluster\"}" >> /etc/ecs/ecs.config

Create the Security Group for the ECS Container Instance:

$ aws --profile default ec2 create-security-group --group-name tools-ecs-sg --description "tools-ecs-sg"
{
    "GroupId": "sg-00000000000000000"
}

Create the Security Group for the Application Load Balancer:

$ aws --profile default ec2 create-security-group --group-name tools-alb-sg --description "tools-alb-sg"
{
    "GroupId": "sg-11111111111111111"
}

Authorize Ingress Ports and Source Addresses of choice:

$ aws --profile default --region eu-west-1 ec2 authorize-security-group-ingress --group-id sg-00000000000000000 --protocol tcp --port 22 --cidr 172.31.0.0/16

$ aws --profile default --region eu-west-1 ec2 authorize-security-group-ingress --group-id sg-00000000000000000 --protocol tcp --port 2049 --cidr 172.31.0.0/16

Create the EBS Mapping for the ECS Container Instance:

$ cat ebs_mapping.json
[
    {
        "DeviceName": "/dev/sda1",
        "Ebs": {
            "DeleteOnTermination": true,
            "VolumeSize": 50,
            "VolumeType": "standard"
        }
    }
]

Deploy the EC2 Instance Deploy Script:

$ cat boot_instance.sh
#!/bin/bash
AWS_PROFILE="default"
AWS_AMI_ID="$(aws --profile default ssm get-parameter --name '/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id' | jq -r '.Parameter.Value')"

AWS_INSTANCE_TYPE="t3a.medium"
AWS_SUBNET_ID="subnet-00000000000000000"
AWS_SSH_KEY="ecs"
AWS_EC2_INSTANCE_COUNT="1"
AWS_SG_ID="sg-00000000000000000"
AWS_INSTANCE_PROFILE_ROLE="arn:aws:iam::xxxxxxxxxxxx:instance-profile/ecs-tools-instance-role"
aws --profile default ec2 run-instances --image-id ${AWS_AMI_ID} --count ${AWS_EC2_INSTANCE_COUNT} \
     --instance-type ${AWS_INSTANCE_TYPE} --key-name ${AWS_SSH_KEY} \
     --subnet-id ${AWS_SUBNET_ID} --security-group-ids ${AWS_SG_ID} \
     --block-device-mappings file://ebs_mapping.json \
     --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ecs-tools-ec2-instance}]' 'ResourceType=volume,Tags=[{Key=Name,Value=ecs-tools-ec2-instance}]' \
     --iam-instance-profile Arn=${AWS_INSTANCE_PROFILE_ROLE} \
     --user-data file://userdata.txt

# to catch ec2_instance_id on run-instances pipe to: | jq -r '.Instances[].InstanceId'

Run the script and create the ECS EC2 Container Instance:

$ $ bash boot_instance.sh
{
    "Groups": [],
    "Instances": [
        {
            "AmiLaunchIndex": 0,
            "ImageId": "ami-0a74b180a0c97ecd1",
            "InstanceId": "i-00000000000000000",
            "InstanceType": "t3a.medium",
            "KeyName": "ecs",
            "LaunchTime": "2020-05-19T11:00:16+00:00",
            "Monitoring": {
                "State": "disabled"
            },
            "Placement": {
                "AvailabilityZone": "eu-west-1a",
                "GroupName": "",
                "Tenancy": "default"
            },
            "PrivateDnsName": "ip-172-31-62-6.eu-west-1.compute.internal",
            "PrivateIpAddress": "172.31.62.6",
            "ProductCodes": [],
            "PublicDnsName": "",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "StateTransitionReason": "",
            "SubnetId": "subnet-00000000000000000",
            "VpcId": "vpc-00000000",
            "Architecture": "x86_64",
            "BlockDeviceMappings": [],
            "ClientToken": "",
            "EbsOptimized": false,
            "Hypervisor": "xen",
            "IamInstanceProfile": {
                "Arn": "arn:aws:iam::xxxxxxxxxxxxx:instance-profile/ecs-tools-instance-role",
                "Id": "X"
            },
            "NetworkInterfaces": [
                {
                    "Attachment": {
                        "AttachTime": "2020-05-19T11:00:16+00:00",
                        "AttachmentId": "eni-attach-xxxxxxxxxxxx",
                        "DeleteOnTermination": true,
                        "DeviceIndex": 0,
                        "Status": "attaching"
                    },
                    "Description": "",
                    "Groups": [
                        {
                            "GroupName": "tools-ecs-sg",
                            "GroupId": "sg-111111111111111111"
                        }
                    ],
                    "Ipv6Addresses": [],
                    "MacAddress": "02:3f:5c:24:05:8a",
                    "NetworkInterfaceId": "eni-xxxxxxxxxxxxx",
                    "OwnerId": "xxxxxxxxxxxxx",
                    "PrivateDnsName": "ip-172-31-62-6.eu-west-1.compute.internal",
                    "PrivateIpAddress": "172.31.62.6",
                    "PrivateIpAddresses": [
                        {
                            "Primary": true,
                            "PrivateDnsName": "ip-172-31-62-6.eu-west-1.compute.internal",
                            "PrivateIpAddress": "172.31.62.6"
                        }
                    ],
                    "SourceDestCheck": true,
                    "Status": "in-use",
                    "SubnetId": "subnet-xxxxxxxxxxxxxxxx",
                    "VpcId": "vpc-xxxxxxxxx",
                    "InterfaceType": "interface"
                }
            ],
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SecurityGroups": [
                {
                    "GroupName": "tools-ecs-sg",
                    "GroupId": "sg-000000000000000000"
                }
            ],
            "SourceDestCheck": true,
            "StateReason": {
                "Code": "pending",
                "Message": "pending"
            },
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "tools-ec2-instance"
                }
            ],
            "VirtualizationType": "hvm",
            "CpuOptions": {
                "CoreCount": 1,
                "ThreadsPerCore": 2
            },
            "CapacityReservationSpecification": {
                "CapacityReservationPreference": "open"
            },
            "MetadataOptions": {
                "State": "pending",
                "HttpTokens": "optional",
                "HttpPutResponseHopLimit": 1,
                "HttpEndpoint": "enabled"
            }
        }
    ],
    "OwnerId": "xxxxxxxxxxxxx",
    "ReservationId": "r-xxxxxxxxxxxxx"
}

Check if instance is running:

$ aws --profile default ec2 describe-instance-status --instance-ids i-0000000000000000
{
    "InstanceStatuses": [
        {
            "AvailabilityZone": "eu-west-1a",
            "InstanceId": "i-0000000000000000",
            "InstanceState": {
                "Code": 16,
                "Name": "running"
            },
            "InstanceStatus": {
                "Details": [
                    {
                        "Name": "reachability",
                        "Status": "passed"
                    }
                ],
                "Status": "ok"
            },
            "SystemStatus": {
                "Details": [
                    {
                        "Name": "reachability",
                        "Status": "passed"
                    }
                ],
                "Status": "ok"
            }
        }
    ]
}

Or, you can just get the status:

$ aws --profile default ec2 describe-instance-status --instance-ids i-0000000000000000 | jq -r '.InstanceStatuses[].InstanceState.Name'
running

ECS Container Instance Checked In

Check that ECS container instance has checked in to the ECS Cluster:

$ aws --profile default ecs list-container-instances --cluster tools-ecs
{
    "containerInstanceArns": [
        "arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:container-instance/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    ]
}

List your ECS Clusters:

$ aws --profile default ecs list-clusters
{
    "clusterArns": [
        "arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:cluster/tools-ecs"
    ]
}

Describe your ECS Cluster:

$ aws --profile default ecs describe-clusters --cluster arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:cluster/tools-ecs
{
    "clusters": [
        {
            "clusterArn": "arn:aws:ecs:eu-west-1:xxxxxxxxxxxx:cluster/tools-ecs",
            "clusterName": "tools-ecs",
            "status": "ACTIVE",
            "registeredContainerInstancesCount": 1,
            "runningTasksCount": 0,
            "pendingTasksCount": 0,
            "activeServicesCount": 0,
            "statistics": [],
            "tags": [],
            "settings": [
                {
                    "name": "containerInsights",
                    "value": "disabled"
                }
            ],
            "capacityProviders": [],
            "defaultCapacityProviderStrategy": []
        }
    ],
    "failures": []
}

To list your Task Definitions:

$ aws --profile default ecs list-task-definitions

Describe a Task Definition:

$ aws --profile default ecs describe-task-definition --task-definition mytaskdefinition

To list the running Tasks in your ECS Cluster:

$ aws --profile prod ecs list-tasks --cluster tools-ecs
{
    "taskArns": []
}

Further Steps: Load Balancers

• Allow the ALB’s SG on the ECS SG (All Traffic) so that the ALB can talk to ECS Container Instances
• Ensure the Roles are included in the KMS policy (usage)
• Ensure the Roles are included in the ECR policy (only for build so this can be scratched)
• Create a Target Group, specify the HTTP port and health checks, dont select instances
• Go to ALB, edit listener, insert rule, host header, forward to, select created target group
• Go to Route53, create new entry, the host that you specified, point the record to the ALB CNAME
• Create service, specify the task definition, add to alb, select existing https listener to point to existing target group

Resources:

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_AWSCLI_EC2.html#AWSCLI_EC2_create_cluster
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
• https://cloudaffaire.com/how-to-create-an-ecs-container-instance-with-ecs-optimized-ami-using-aws-cli/
• https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskDefinition.html
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
• https://aws.amazon.com/premiumsupport/knowledge-center/ecs-create-docker-volume-efs/