Tuesday, 17 January 2017

Elasticsearch backup using snapshot

The snapshot and restore module allows to create snapshots of individual indices or an entire cluster into a remote repository like shared file system.These snapshots are great for backups because they can be restored relatively quickly but they are not archival because they can only be restored to versions of Elasticsearch that can read the index.Its very easy to take backup of your elasticsearch index.you can just make a snapshot, rsync/NFS export the files to another host and restore them from those files.

Elasticsearch backup using snapshot

Snapshot repository : Register a snapshot repository in Elasticsearch.

curl -XPUT 'http://localhost:9200/_snapshot/backup' -d '{
  "type": "fs",
  "settings": {
    "location": "/home/wantcode/Documents/es/backup",
    "compress": true

Note : If path.repo is not set in your elasticsearch.yml(/etc/elasticsearch/elasticsearch.yml) configuration please set it.
path.repo = /home/wantcode/Documents/es/backup

Also set the permission for /home/wantcode/Documents/es/backup directory.
chown elasticsearch:elasticsearch /home/wantcode/Documents/es/backup.

Now restart your elastics server.(root user command otherwise add sudo)

service elasticsearch restart

After register the snapshot . Make a script for taking snapshot using script (takesnapshot.sh).

SNAPSHOT=`date +%Y%m%d-%H%M%S`
curl -XPUT "localhost:9200/_snapshot/backup/$SNAPSHOT?wait_for_completion=true"

It’s very easy to set up this backup, there is currently no way included to remove old snapshots.so you can write a small script that keeps the last 30 snapshots and deletes anything older.


# The amount of snapshots we want to keep.

# Name of our snapshot repository

# Get a list of snapshots that we want to delete
SNAPSHOTS=`curl -s -XGET "localhost:9200/_snapshot/$REPO/_all" \
  | jq -r ".snapshots[:-${LIMIT}][].snapshot"`

# Loop over the results and delete each snapshot
 echo "Deleting snapshot: $SNAPSHOT"
 curl -s -XDELETE "localhost:9200/_snapshot/$REPO/$SNAPSHOT?pretty"
echo "Done!"

Now you can check your /home/wantcode/Documents/es/backup folder all index folders are created so your backup is created over the folder. Now you can try to delete the some index and restore it using the script.

Get the list of snapshot using command.

curl -s -XGET "localhost:9200/_snapshot/backup/_all?pretty"

Output like :

  "snapshots" : [ {
    "snapshot" : "20170117-131953",
    "version_id" : 2010099,
    "version" : "2.1.0",
    "indices" : ["jdbc", "index_2111", "river", "catalog" ],
    "state" : "SUCCESS",
    "start_time" : "2017-01-17T07:49:53.817Z",
    "start_time_in_millis" : 1484639393817,
    "end_time" : "2017-01-17T07:51:05.626Z",
    "end_time_in_millis" : 1484639465626,
    "duration_in_millis" : 71809,
    "failures" : [ ],
    "shards" : {
      "total" : 25,
      "failed" : 0,
      "successful" : 25
  } ]

Now make a script to restore a particular index (snapshotrestore.sh).

# Restore a snapshot from our repository directory

# We need to close the index first
curl -XPOST "localhost:9200/catalog/_close"

# Restore the snapshot we want
curl -XPOST "http://localhost:9200/_snapshot/backup/$SNAPSHOT/_restore" -d '{
 "indices": "catalog"

# Re-open the index
curl -XPOST 'localhost:9200/catalog/_open'

After run this script it will restore the catalog index in your elasticsearch server.please comment for any further help.

Thursday, 5 January 2017

International PHP Conference 2017

The International PHP Conference is the world's first PHP conference and stands since more than a decade for top-notch pragmatic expertise in PHP and web technologies. At the IPC, internationally renowned experts from the PHP industry meet up with PHP users and developers from large and small companies.....Read More


Highlights : 

Expo with exiting exhibitors
Cool PHP & the Gang: 60+ international top speakers
90+ best practice sessions
Official certificate for attendees
Conference Combo: Visit the webinale For Free
All inclusive: Changing buffets, snacks & refreshing drinks

Featured post

How to create dynamic schema in mongodb

Sometime what we need in our application is to create mongodb schema for collection and some of our collection schema are same but names are...

Popular Posts