AWS Elasticsearch Register S3 Repository for Snapshots using the CLI

In order to make a manual Snapshot in Amazon's Elasticsearch Service, we need to create a S3 repository where the data will reside.

In order to do this, we need to sign the request with an IAM role that grants permissions to Amazon ES. We will sign the request using Python.

Overview:

We will create the following:

• S3 Bucket: my-es-snapshot-repo
• IAM Role: es-snapshots-role
• IAM Policy: es-snapshot-access
• Python script to sign the request: sign.py

Create S3 Bucket:

First we will create a S3 bucket for our snapshots:

$ aws s3 mb s3://my-es-snapshot-repo --region eu-west-1

IAM: Create a Role and attach a Trust Relationship Policy:

Save the following policy as trust-relationship-es.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

$ aws iam create-role --role-name es-snapshots-role --assume-role-policy-document file://trust-relationship-es.json

Output:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "Service": "es.amazonaws.com"
                    },
                    "Effect": "Allow",
                    "Sid": ""
                }
            ]
        },
        "RoleId": "ABCDEFGHIJKLM1234567",
        "CreateDate": "2016-09-13T08:13:43.623Z",
        "RoleName": "es-snapshots-role",
        "Path": "/",
        "Arn": "arn:aws:iam::123456789012:role/es-snapshots-role"
    }
}

Save the following policy as: es-snapshot-policy.json

{
    "Version":"2012-10-17",
    "Statement":[
        {
            "Action":[
                "s3:ListBucket"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::my-es-snapshot-repo"
            ]
        },
        {
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "iam:PassRole"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::my-es-snapshot-repo/*"
            ]
        }
    ]
}

Create a IAM Policy and attach the created policy document:

$ aws iam create-policy --policy-name es-snapshot-access --policy-document file://es-snapshot-policy.json

== Output: ==

{
    "Policy": {
        "PolicyName": "es-snapshot-access",
        "CreateDate": "2016-09-13T08:14:17.759Z",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "PolicyId": "ABCDEFGHJKL12345",
        "DefaultVersionId": "v1",
        "Path": "/",
        "Arn": "arn:aws:iam::123456789012:policy/es-snapshot-access",
        "UpdateDate": "2016-09-13T08:14:17.759Z"
    }
}

Now, attach the policy to the role:

$ aws iam attach-role-policy --policy-arn arn:aws:iam::123456789012:policy/es-snapshot-access --role-name es-snapshots-role

Python: Signing the request:

Note: For this tutorial, I was using my AWS Credential Provider for getting my access keys, but if not, you need pass AWS Credentials like the following:

client = ESConnection(
    region='eu-west-1',
    host='search-domain-name',
    aws_access_key_id='my-access-key-id',
    aws_secret_access_key='my-access-key',
    is_secure=False)

If you have multiple profiles in your credential provider, you can set the profile using profile_name. If you are using the default profile, you can remove the profile_name property.

Save the following script as: sign.py

from boto.connection import AWSAuthConnection
 
class ESConnection(AWSAuthConnection):
 
    def __init__(self, region, **kwargs):
        super(ESConnection, self).__init__(**kwargs)
        self._set_auth_region_name(region)
        self._set_auth_service_name("es")
 
    def _required_auth_capability(self):
        return ['hmac-v4']
 
if __name__ == "__main__":
 
    client = ESConnection(
            region='eu-west-1',
            host='search-domain.eu-west-1.es.amazonaws.com',
            profile_name='ifOtherThanDefault',
            is_secure=False)
 
    print 'Registering Snapshot Repository'
    resp = client.make_request(method='PUT',
            path='/_snapshot/es-index-backups',
            data='{"type": "s3","settings": { "bucket": "my-es-snapshot-repo","region": "eu-west-1","role_arn": "arn:aws:iam::123456789012:role/es-snapshots-role"}}')
    body = resp.read()
    print body

Now execute the script, and if everything is good, you should be returned with {"acknowledged":true}:

$ python sign.py
Registering Snapshot Repository
{"acknowledged":true}

Creating the snapshot:

curl -XPUT 'search-domain.eu-west-1.es.amazonaws.com/_snapshot/es-index-backups/mysnapshot'
{"accepted":true}

Now your data should be backed up, you can verify by listing the S3 repository:

$ aws s3 ls s3://rb-es-snapshots/indices/
                           PRE .kibana-4/
                           PRE profiles/
                           PRE logstash-2016.09.08/
                           PRE logstash-2016.09.07/

For more information on this, please see the AWS Documentation

​