Getting Started with AWS Elasticsearch Service

Elasticsearch - What is it?

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Elasticsearch is the most popular enterprise search engine followed by Apache Solr, also based on Lucene
-- Source

AWS Managed Elasticsearch service:

Amazon Web Services offers a managed Elasticsearch service. WIth this offering it makes it easy to launch, operate and to scale Elasticsearch into the AWS Cloud.

Benefits:

• Simple to Deploy
• Easy To Administer
• Scalable
• Comes integrated with Logstash and Kibana
• Cost Affective
• Secure

More information can be found here

Getting started with AWS Elasticsearch Service:

We will go through a quick setup on how to create your search domain and the basic usage thereof.

Creating a Search Domain:

Note: We will need the aws cli tools in order to use the cli.

We will create a ES Domain consisting:

• Elasticsearch Domain called: logtest
• Instance Type: 2 Instances of m3.medium.elasticsearch
• Storage: 100GB Magnetic EBS Volume per node
• Security: Allow access from 52.30.142.186

Create the Elasticsearch Search Domain:

aws es create-elasticsearch-domain --domain-name weblogs \
--elasticsearch-cluster-config InstanceType=m3.medium.elasticsearch,InstanceCount=2 \
--ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100 \
--access-policies '
{"Version": "2012-10-17", 
    "Statement":
        [{"Action": 
            "es:*", 
            "Principal":"*",
            "Effect": "Allow", 
            "Condition": 
                {"IpAddress":
                {"aws:SourceIp":
                    ["192.0.2.0/32"]
                }
            }
        }
    ]
}
'

We can also go a lot deeper into locking down security. There are 3 ways of using Access Configuration, they are:

• Resource-Based Access Policies
• IP-Based Policy
• IAM User and Role-Based Policies

More on that is covered in detail over here

Describe your ES Domain via CLI

We will use jq to redirect the output in order to to provide us with the endpoint (optional)

$ wget -O /sbin/jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
$ chmod +x /sbin/jq

Let's get our Endpoint Address:

aws es describe-elasticsearch-domain --domain-name weblogs | jq .DomainStatus.Endpoint

Output:

"search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com"

Basic Usage Examples:

Endpoint Check:

$ curl 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com'

Check Cluster Health:

curl -XGET 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_cluster/health?pretty=true'

Endpoint Check:

curl -XPUT "https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/movies/movie/1" -d' {

   "title": "Deadpool",
   "director": "Tim Miller",
   "year": 2016

}' 

Querying All the Content:

curl -XGET 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_search?pretty=true'

Query All the Nodes configured on the Elasticsearch Cluster:

curl -XGET 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_nodes'   | python -m json.tool |more

List All Indexes:

curl 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_cat/indices?v'

View the Mappings:

curl -XGET 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_all/_mapping' | python -m json.tool |more

Deleting Indexes:

curl -XDELETE 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_all/'

Adding Data:

curl -XPUT "https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/movies/movie/2" -d' {

   "title": "Batman vs Superman",
   "director": "Zack Snyder",
   "year": 2016

}' 

Loading Data from JSON:

curl -XPUT 'https://search-weblogs-trofnbngj6fqk3rkbawcsvmqwu.eu-west-1.es.amazonaws.com/_bulk' --data-binary @bulkdata.json

Migrating from a Local Elasticsearch cluster to AWS Elasticsearch cluster coming soon and will be added here when I have completed it.