Sysadmins

Snapshot and Restore Indices on Elasticsearch

Ruan Bekker -

This is a demonstration of how to create snapshots (hosted on AWS S3) of indices on elasticsearch, and also how to restore from those snapshots.

For more  info from Elasticsearch Documentation:

Create a Index

Create the index that we will use to snapshot:

$ curl -XPUT -H 'Content-Type: application/json' 'https://my-es.eu-west-1.es.amazonaws.com/ruan-test-2020.02.21'

Ingest a document into the index that we created:

$ curl -XPUT -H 'Content-Type: application/json' 'https://my-es.eu-west-1.es.amazonaws.com/ruan-test-2020.02.21/docs/1' -d
'
{
    "name": "ruan",
    "surname": "bekker"
}
'

Let's have a look at our indices endpoint:

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_cat/indices/ruan*?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open ruan-test-2020.02.21 bkZIx2m2QA6kU2tntoLbRQ 5 1 1 0 10.2kb 5.1kb

Snapshot Repositories

If you have not created a S3 Snapshot Repository, you can follow this post.

Let's have a look at our available snapshot repositories. If you are using elasticsearch service on AWS, you will see a cs-automated repository, which will be used by Amazon for automated snapshots.

The snapshot repository that we are interested in will be es-index-backups :

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_cat/repositories?v'
id                type
cs-automated      s3
es-index-backups  s3

Let's describe the snapshot repository:

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_snapshot/es-index-backups?pretty'
{
    "es-index-backups" : {
    "type" : "s3",
        "settings" : {
            "bucket" : "my-es-snapshot-repo",
            "region" : "eu-west-1",
            "role_arn" : "arn:aws:iam::xxxxxxxxxxxx:role/es-snapshots-role"
        }
    }
}

Create a Snapshot

Let's create a snapshot named: mysnapshot_ruan-test-2020.02.21_1 of the index ruan-test-2020.02.21 which will be stored on S3:

$ curl -XPUT -H 'Content-Type: application/json' 'https://my-es.eu-west-1.es.amazonaws.com/_snapshot/es-index-backups/mysnapshot_ruan-test-2020.02.21_1?wait_for_completion=true&pretty=true' -d '
{
    "indices": "ruan-test-2020.02.21",
    "ignore_unavailable": true,
    "include_global_state": false
}'

Response:

{
    "snapshot" : {
        "snapshot" : "mysnapshot_ruan-test-2020.02.21_1",
        "uuid" : "YRTE5922QCeqyEaMxPqb1A",
        "version_id" : 6000199,
        "version" : "6.0.1",
        "indices" : [ "ruan-test-2020.02.21" ],
        "state" : "SUCCESS",
        "start_time" : "2020-02-21T13:20:11.497Z",
        "start_time_in_millis" : 1527254411497,
        "end_time" : "2020-02-21T13:20:11.886Z",
        "end_time_in_millis" : 1527254411886,
        "duration_in_millis" : 389,
        "failures" : [ ],
        "shards" : {
        "total" : 5,
        "failed" : 0,
        "successful" : 5
        }
    }
}
Snapshot Response

We can also see if any snapshots are running:

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_snapshot/es-index-backups/_status?pretty'
{
    "snapshots" : [ ]
}

Verify that the snapshot was created:

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_cat/snapshots/es-index-backups?v&s=id'
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
mysnapshot_ruan-test-2020.02.21_1 SUCCESS 1527254411 06:20:11 1527254411 06:20:11 389ms 1 5 0 5

Restore a Snapshot

Let's view the metadata of our snapshot:

$ curl -XGET 'https://my-es.eu-west-1.es.amazonaws.com/_snapshot/es-index-backups/mysnapshot_ruan-test-2020.02.21_1?pretty'
{
    "snapshots" : [ {
    "snapshot" : "mysnapshot_ruan-test-2020.02.21_1",
    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
    "version_id" : 6000199,
    "version" : "6.0.1",
    "indices" : [ "ruan-test-2020.02.21" ],
    "state" : "SUCCESS",
    "start_time" : "2020-02-21T13:20:11.497Z",
    "start_time_in_millis" : 1527254411497,
    "end_time" : "2020-02-21T13:20:11.886Z",
    "end_time_in_millis" : 1527254411886,
    "duration_in_millis" : 389,
    "failures" : [ ],
    "shards" : {
        "total" : 5,
        "failed" : 0,
        "successful" : 5
        }
    } ]
}

We can view the objects on S3 as well:

$ aws s3 --profile personal ls s3://my-es-snapshot-repo/ | grep YRTE5922QCeqyEaMxPqb1A
2020-02-21 15:20:12 90 meta-YRTE5922QCeqyEaMxPqb1A.dat
2020-02-21 15:20:12 258 snap-YRTE5922QCeqyEaMxPqb1A.dat

We will be restoring our index from a snapshot, so let's go ahead and delete the index:

$ curl -XDELETE -H 'Content-Type: application/json' 'https://my-es.eu-west-1.es.amazonaws.com/ruan-test-2020.02.21'

Restore the index from our snapshot:

$ curl -XPOST -H 'Content-Type: application/json' 'https://my-es.eu-west-1.es.amazonaws.com/_snapshot/index-backups/mysnapshot_ruan-test-2020.02.21_1/_restore -d '
{
    "indices": "ruan-test-2020.02.21",
    "ignore_unavailable": true,
    "include_global_state": false,
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
}'

Thank You

I hope this was useful, I'd like to hear from you, reach out to me on Twitter: @ruanbekker.