Elasticsearch cluster with Snapshots in Vagrant

With this code files, we will install and configure Elasticsearch cluster with 4 nodes. Master node elk and slave nodes elknd1, elknd2, elknd3

When cluster will be UP and ready scripts will download some sample databases from Internet to the elknd3 node and through master elk (Actually there is no difference between nodes and we can use any of the nodes to upload) API upload them inside of the Elasticsearch. In the Elasticsearch databases named as the Indices. Scripts will check Indices count if count will be as needed then it will configure Snapshot for the all nodes. For that in the master (elk) node will be configured NFS server and this configured folder will be mounted to the other (elknd1, elknd2, elknd3) nodes. In the end, the snapshot will be configured through API and executed to get the first one.

Vagrant Virtual machines:

elk
elknd1
elknd2
elknd3

The content of the Vagrantfile as follows:

# cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

ip = "192.168.120.40"

current_dir = File.dirname(File.expand_path(__FILE__))
Vagrant.configure("2") do |elk|
  require './vagrant-provision-reboot-plugin'
  elk.vm.box = "centos/7"
  elk.vm.define "elk" do |subelk|
    subelk.vm.network :private_network, ip: ip
    subelk.vm.hostname = "elk"
    subelk.ssh.forward_agent = true
    subelk.vm.provider :virtualbox do |v1|
      v1.customize ["modifyvm", :id, "--memory", 2048]
      v1.customize ["modifyvm", :id, "--name", "elk"]
    end
    subelk.vm.provision "shell", path: "scripts/install.sh"
    subelk.vm.provision "shell", path: "scripts/elkinstall.sh", args: ip
    subelk.vm.provision "shell", path: "scripts/nfssrvconf.sh", args: ip
    subelk.vm.provision :unix_reboot
  end
  (1..3).each do |i|
     elk.vm.define "elknd#{i}" do |elkndconfig|
       elkndconfig.vm.network :private_network, ip: "192.168.120.2#{i}"
       elkndconfig.vm.hostname = "elknd#{i}"
       elkndconfig.ssh.forward_agent = true
       elkndconfig.vm.network :forwarded_port, guest: 22, host: "2002#{i}", id: "ssh"
       elkndconfig.vm.provider :virtualbox do |v1|
         v1.customize ["modifyvm", :id, "--memory", 1024]
         v1.customize ["modifyvm", :id, "--name", "elknd#{i}"]
       end
       elkndconfig.vm.provision "shell", path: "scripts/install.sh"
       elkndconfig.vm.provision "shell", path: "scripts/elknodes.sh", args: "#{i}"
       elkndconfig.vm.provision :unix_reboot
       elkndconfig.vm.provision "shell", path: "scripts/importDataGetSnapshot.sh", args: ip
     end
  end
end

From the previous Vagrantfile as we see the elk node IP will be 192.168.120.40 and install.sh, elkinstall.sh, nfssrvconf.sh provisioning codes after deployment. For the elknd1, elknd2, elknd3 provisioning codes will be install.sh and elknodes.sh scripts.

Let’s start analyzing provision codes. Firstly, we start to describe the install.sh script because it is used for all servers.

The following script installs all needed components and Oracle Java to the Linux.

$ cat scripts/install.sh
#!/usr/bin/env bash

sudo yum -y install epel-release && sudo yum -y install bash-completion net-tools bind-utils wget telnet vim expect gcc-c++ make jq unzip nfs-utils
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jre-8u181-linux-x64.rpm"
sudo yum localinstall -y /home/vagrant/jre-8u181-linux-x64.rpm
sudo sed -i.bak -e 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd
sudo systemctl stop firewalld && sudo systemctl disable firewalld
sudo echo 'export JAVA_HOME=/usr/java/jre1.8.0_181-amd64/bin/java' >> ~/.bashrc
sudo echo 'export JAVA_HOME=/usr/java/jre1.8.0_181-amd64/bin/java' >> /home/vagrant/.bashrc
sudo sed -i.bak -e 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
sudo sed -i.bak -e 's/keepcache=0/keepcache=1/' /etc/yum.conf

Now we will explain elk master node provisioning scripts. In the elkinstall.sh script will be configured official Elasticsearch repository. Then will be installed Elasticsearch (with cluster configuration), Kibana and Logstash. In the end, all services will be started and enabled.

$ cat scripts/elkinstall.sh
#!/usr/bin/env bash

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

sudo cat < /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo yum install -y elasticsearch
sudo cp /etc/elasticsearch/elasticsearch.yml /root
sudo cat  ~/.bashrc
#sudo echo 'export PATH=$PATH:/usr/share/logstash/bin' >> /etc/environment
sudo cp /etc/logstash/logstash.yml /root/
sudo cat < /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
EOF
sudo systemctl start logstash && sudo systemctl enable logstash

With the scripts/nfssrvconf.sh script will be configured NFS server to the SNAPSHOT folder with access only elknd1, elknd2, and elknd3. Service will be started and enabled.

$ cat scripts/nfssrvconf.sh
#!/usr/bin/env bash

sudo cat < /etc/exports
/etc/elasticsearch/elasticsearch-backup 192.168.120.21(rw,sync,no_root_squash)
/etc/elasticsearch/elasticsearch-backup 192.168.120.22(rw,sync,no_root_squash)
/etc/elasticsearch/elasticsearch-backup 192.168.120.23(rw,sync,no_root_squash)
EOF

sudo systemctl restart nfs-server && sudo systemctl enable nfs-server

Let’s start to explain elknd1, elknd2, and elknd3 servers provisioning. The script scripts/elknodes.sh configure Elasticsearch repository, install package and configure needed files. Then start and enable service to all nodes.

$ cat scripts/elknodes.sh
#!/usr/bin/env bash

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

sudo cat < /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo yum install -y elasticsearch
sudo cp /etc/elasticsearch/elasticsearch.yml /root
sudo cat < /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: ["/etc/elasticsearch/elasticsearch-backup"]
cluster.name: elkepam
node.name: node-node$1
node.master: false
node.data: true
network.host: 192.168.120.2$1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.120.40", "192.168.120.21","192.168.120.22","192.168.120.23"]
#discovery.zen.minimum_master_nodesdiscovery.zen.minimum_master_nodes: 2
EOF

sudo touch /etc/elasticsearch/elasticsearch.keystore
sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/
sudo chmod -R 750 /etc/elasticsearch/
sudo systemctl enable elasticsearch && sudo systemctl start elasticsearch

Script scripts/importDataGetSnapshot.sh mount exported NFS disk to the needed PATH for the snapshots. Then will check if OS hostname is elknd3 and cluster up and running then, download some databases and import to the Elasticsearch through API. After that checking indices count and if it is right it will configure Snapshot through API and start it. All snapshots will be stored in the /etc/elasticsearch/elasticsearch-backup folder.

$ cat scripts/importDataGetSnapshot.sh
#!/usr/bin/env bash

cat  /etc/fstab
$1:/etc/elasticsearch/elasticsearch-backup /etc/elasticsearch/elasticsearch-backup nfs defaults 0 0
EOF

sudo mount -a

if [ "$(hostname)" != "elknd3" ]
then
    echo This node most be the last one!
    exit 0
fi

while [ "$(curl -s -XGET "http://$1:9200/_cluster/state?pretty" | jq '.nodes' | grep 9300 | wc -l)" != "4" ]
do
    sleep 3
    echo "Waiting till full cluster environment will be up and running!"
done

wget https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json
wget https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip
unzip accounts.zip
wget https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz
gunzip logs.jsonl.gz

curl -s -H "Content-Type: application/json" -XPOST http://$1:9200/shakespeare/doc/_bulk?pretty --data-binary @shakespeare_6.0.json
curl -s -H "Content-Type: application/json" -XPOST http://$1:9200/_bulk?pretty --data-binary @logs.jsonl
curl -s -H "Content-Type: application/json" -XPOST http://$1:9200/bank/account/_bulk?pretty --data-binary @accounts.json

indexCount=$(curl -s -H "Content-Type: application/json" -XGET http://$1:9200/_cat/indices?v | grep -v index | awk '{ print $3 }' | wc -l)

if [ "$indexCount" -gt "3" ]
then
    echo Cluster successfully imported all data files.
fi

curl -s -H 'Content-Type: application/json' -XPUT "http://$1:9200/_snapshot/all-backup" -d '
{
   "type": "fs",
   "settings": {
       "compress" : true,
       "location": "/etc/elasticsearch/elasticsearch-backup"
   }
}'

curl -s -H 'Content-Type: application/json' -XPUT "http://$1:9200/_snapshot/all-backup/snapshot-number-one?wait_for_completion=true"
curl -s -H 'Content-Type: application/json' -XGET "http://$1:9200/_snapshot/_all""

vagrant-provision-reboot-plugin.rb is the plugin file for the vagrant which will be used to reboot after deployment.

To execute the environment just download all code files from Git repository and run Vagrant.

# git clone https://github.com/jamalshahverdiev/vagrant-elasticsearch-cluster-with-snapshot.git
# cd vagrant-elasticsearch-cluster-with-snapshot && vagrant up

To restore Snapshot from CLUSTER1 to the CLUSTER2 just configure CLUSTER2(Snapshot must be configured via API but not executed) as the CLUSTER1 and execute the following steps. In the CLUSTER1 archive content of the /etc/elasticsearch/elasticsearch-backup folder to the elksnapshot.tar.gz file:

# cd /etc/elasticsearch/elasticsearch-backup && tar -zcvf elksnapshot.tar.gz .

In the CLUSTER2 upload the archived elksnapshot.tar.gz file from CLUSTER1 to the /etc/elasticsearch/elasticsearch-backup folder and then extract it:

# cp /root/elksnapshot.tar.gz /etc/elasticsearch/elasticsearch-backup
# cd /etc/elasticsearch/elasticsearch-backup && tar -zxvf elksnapshot.tar.gz

Configure snapshot PATH in the cluster:

# curl -s -H 'Content-Type: application/json' -XPUT "http://192.168.120.40:9200/_snapshot/all-backup" -d '
{
   "type": "fs",
   "settings": {
       "compress" : true,
       "location": "/etc/elasticsearch/elasticsearch-backup"
   }
}'

Look at the snapshot which you created before and extracted arcive from previous Snapshot (As we can see we have 5 indices, 3 logstash, 1 bank and 1 Shakespeare):

[root@elkmaster elasticsearch-backup]# curl -s -H 'Content-Type: application/json' -XGET "http://192.168.120.40:9200/_snapshot/all-backup/_all" | jq
{
  "snapshots": [
    {
      "snapshot": "snapshot-number-one",
      "uuid": "M-I1bd-YT22V6MgEhBvbJA",
      "version_id": 6040099,
      "version": "6.4.0",
      "indices": [
        "logstash-2015.05.18",
        "logstash-2015.05.20",
        "bank",
        "logstash-2015.05.19",
        "shakespeare"
      ],
      "include_global_state": true,
      "state": "SUCCESS",
      "start_time": "2018-09-12T07:06:59.188Z",
      "start_time_in_millis": 1536736019188,
      "end_time": "2018-09-12T07:07:04.184Z",
      "end_time_in_millis": 1536736024184,
      "duration_in_millis": 4996,
      "failures": [],
      "shards": {
        "total": 25,
        "failed": 0,
        "successful": 25
      }<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
    }
  ]
}

If you try to get list of the indices it will be empty:

# curl -s -H 'Content-Type: application/json' -XGET "http://192.168.120.40:9200/_cat/indices"

Restore snapshot-number-one which we created in the CLUSTER1:

# curl -s -H 'Content-Type: application/json' -XPOST "http://192.168.120.40:9200/_snapshot/all-backup/snapshot-number-one/_restore"
{"accepted":true}

Look at the indices count:

# curl -s -H 'Content-Type: application/json' -XGET "http://192.168.120.40:9200/_cat/indices"
green open shakespeare         pq4620uoQOiXXC-lQKwi-Q 5 1 111396 0  45.8mb 22.9mb
green open logstash-2015.05.18 QlFO8vHxSPyBBZ0JbKa0mw 5 1   4631 0  51.5mb 25.7mb
green open logstash-2015.05.20 4LxKjqnVTMaKI_QYzZIycA 5 1   4750 0  47.7mb 26.1mb
green open logstash-2015.05.19 TPoE-3SdRiqDmPHGs2MLIQ 5 1   4624 0  48.1mb   25mb
green open bank                tgU6Om8pRfKklNvzo9riaA 5 1   1000 0 950.1kb  475kb

Look at the selected Snapshot:

[root@elkmaster elasticsearch-backup]# curl -s -H 'Content-Type: application/json' -XGET "http://192.168.120.40:9200/_snapshot/all-backup/snapshot-number-one" | jq
{
  "snapshots": [
    {
      "snapshot": "snapshot-number-one",
      "uuid": "M-I1bd-YT22V6MgEhBvbJA",
      "version_id": 6040099,
      "version": "6.4.0",
      "indices": [
        "logstash-2015.05.18",
        "logstash-2015.05.20",
        "bank",
        "logstash-2015.05.19",
        "shakespeare"
      ],
      "include_global_state": true,
      "state": "SUCCESS",
      "start_time": "2018-09-12T07:06:59.188Z",
      "start_time_in_millis": 1536736019188,
      "end_time": "2018-09-12T07:07:04.184Z",
      "end_time_in_millis": 1536736024184,
      "duration_in_millis": 4996,
      "failures": [],
      "shards": {
        "total": 25,
        "failed": 0,
        "successful": 25
      }
    }
  ]
}