~Note:~ This post is old and is scheduled to be updated.
Centralized logging, analytics and visualization with ElasticSearch, Filebeat, Kibana and Logstash.
Our ELK Stack will consist of:
Elasticsearch:
Stores all of the logs
Kibana: Web interface for searching and visualizing logs
Logstash: The server component of Logstash that processes incoming logs
Filebeat: Installed on our client servers that will push their logs to Logstash
Let's get started on setting up our environment for our ELK Stack.
Install Java 8:
$ cd ~
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm"
$ yum localinstall jdk-8u65-linux-x64.rpm
$ rm -rf jdk-8u65-linux-x64.rpm
Install Elasticsearch:
$ rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
$ vi /etc/yum.repos.d/elasticsearch.repo
Add our repo configuration:
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Then install elasticsearch:
$ yum -y install elasticsearch
Configure Elasticsearch:
$ vi /etc/elasticsearch/elasticsearch.yml
We will set our network.host to localhost
network.host: localhost
Start elasticsearch and enable on startup:
$ service elasticsearch start
$ chkconfig elasticsearch on
Install Kibana:
Create repo configuration for Kibana:
$ vi /etc/yum.repos.d/kibana.repo
[kibana-4.4]
name=Kibana repository for 4.4.x packages
baseurl=http://packages.elastic.co/kibana/4.4/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
Install Kibana:
$ yum -y install kibana
Configure Kibana:
$ vi /opt/kibana/config/kibana.yml
We will set our server.host to localhost:
server.host: "localhost"
Start kibana and enable on startup:
$ service kibana start
$ chkconfig kibana on
Install Logstash:
Create repo configuration for Logstash:
$ vi /etc/yum.repos.d/logtash.repo
[logstash-2.2]
name=logstash repository for 2.2 packages
baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
Install Logstash:
$ yum -y install logstash
Generate SSL Certificates:
$ vi /etc/pki/tls/openssl.cnf
Find the configuration line, and replace your private servers ip address:
subjectAltName = IP: your_servers_private_ip_here
After saving your config, we will generate the SSL Certificate and Private Key:
$ cd /etc/pki/tls
$ openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Keep in mind that all the servers that will be pushing logs to logstash, needs to have the logstash-forwarder.crt certificate.
Configure logstash:
$ vi /etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
$ vi /etc/logstash/conf.d/10-syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
$ vi /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
$ service logstash configtest
Note:
If you are using elasticsearch v2+, then change host to hosts in this following configuration:
output {
elasticsearch { hosts => localhost }
stdout { codec => rubydebug }
}
If all is okay, restart logstash:
$ service logstash restart
We will load Kibana dashboards and Beats index patterns that is provided by Elastic that can help us to get started with Kibana:
cd ~
curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
unzip beats-dashboards-*.zip
cd beats-dashboards-*
./load.sh
These are our Index Patterns that we just loaded:
[packetbeat-]YYYY.MM.DD
[topbeat-]YYYY.MM.DD
[filebeat-]YYYY.MM.DD
[winlogbeat-]YYYY.MM.DD
We are using Filebeat to ship logs to Elasticsearch, we should then load a Filebeat index template:
cd ~
curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
```<p>
Now we will load our template into Elasticsearch:
$ curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' [email protected]
The expected output should look like this:
```json
{
"acknowledged" : true
}
Set Up Filebeat on our Client Servers:
Copy our certificate onto our client servers:
$ scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_IP_address:/tmp
On our client servers:
[clientserver] $ mkdir -p /etc/pki/tls/certs
[clientserver] $ cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/
[clientserver] $ rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
Create repo configuration for beats on our client servers:
$ vi /etc/yum.repos.d/elastic-beats.repo
[beats]
name=Elastic Beats Repository
baseurl=https://packages.elastic.co/beats/yum/el/$basearch
enabled=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
gpgcheck=1
Install filebeat on our client servers:
[clientserver] $ yum -y install filebeat
Configure Filebeat:
[clientserver] $ vi /etc/filebeat/filebeat.yml
...
-
paths:
- /var/log/secure
- /var/log/messages
- /var/log/squid/*.log
...
document_type: syslog
...
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["your-ELK-private-ip:5044"]
...
bulk_max_size: 1024
...
tls:
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
[clientserver] $ service filebeat start
Test filebeat installation:
[clientserver] $ curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
```<p>
Expected output:
```json
...
{
"_index" : "filebeat-2016.02.19",
"_type" : "log",
"_id" : "AVL5Dl3hhgXUIOAm_McP",
"_score" : 1.0,
"_source" : {
"@metadata" : {
"beat" : "filebeat",
"type" : "log"
},
"@timestamp" : "2016-02-19T10:23:33.447Z",
"beat" : {
"hostname" : "elk.int.ruanbekker.com",
"name" : "elk.int.ruanbekker.com"
},
"count" : 1,
"fields" : null,
"input_type" : "log",
"message" : "2016/02/19 12:03:38| Loaded Icons.",
"offset" : 1811,
"source" : "/var/log/squid/caching.log",
"type" : "log"
}
...
Finally, Connect to Kibana Web Interface on http://your_endpoint:5061
Comments