Storing Elasticsearch Data on a Separate Ubuntu Partition

By default on Ubuntu Server, Elasticsearch 6.6.0 is installed to the /var/lib/elasticsearch directory on the partition your OS resides on; the system partition. If you were to have an ELK cluster running in a production environment with 100+ endpoints feeding thousands of logs everyday, you may start to find that your Elasticsearch nodes’ drives are getting rather large, or even almost full. This is due to the fact that Elasticsearch is the place where ALL indices are stored, meaning the plethora of information you see in Kibana is, no, not magic. With all of this data stored on the main system partition, if the drive were to fill up it could freeze the OS and take the entire node with it. This seems rather important to avoid, as it would be a real shame to watch your cluster crumble due to the fact that you simply had too much information. Luckily, there is a solution; simply add another drive to your Ubuntu Server Machine or VM, partition it, then configure Elasticsearch to store data there instead. Below is a guide on how to do just that.

What do I need?

• Functioning ELK Stack (Single or Multi-Node)
• Elasticsearch running on Ubuntu Server Machines/VMs
• • NOTE: This guide uses Ubuntu Server 18.04
• A new hard drive (physical or virtual) to attach and mount

Step 1: Attach new drive to your Elasticsearch Ubuntu Machine/VM

This seems rather obvious, but you do need to attach that new drive before we can get going. So, yeah. Attach that.

If you are using a VM like myself, this process can be done by going Edit Settings –> Add New Device –> Hard Disk as shown below:

Once your machine is booted up, run the following command to ensure the drive is recognized by the system:

fdisk -l

You should see an unknown drive, most likely labeled along the lines of /dev/sdb. Take note of this, as this is the drive you need to target for formatting later.

Step 2: Partition and Format the new drive

To start the formatting process, use fdisk to target the new drive by the name you identified earlier (i.e. /dev/sdb)

fdisk /dev/sdb

Now do the following:

1. Press N for new partition
2. Press ENTER to go with ALL of the defaults
3. Press P to show partition table and ensure status
4. Press W to write to disk
5. Press Q to exit (if necessary)

Running fdisk -l again should show your new partition:

The default file system for Ubuntu Server is ext4. In this case, we want to keep that same file system for the sake of system uniformity. Use the following command to format your new drive partition:

mkfs.ext4 /dev/sdb1

Step 3: Mount the new partition to Ubuntu

Ubuntu requires that a partition be mounted to a designated directory, usually found within /mnt. For the sake of this guide, I created /mnt/es_data to mount my new partition to:

mkdir /mnt/es_data

To avoid having to remount this drive every time your Ubuntu system initializes, you can edit the following file with pre-designated instructions for auto-mounting:

sudo nano /etc/fstab

Add the following information to the file, substituting my partition and mount directory for yours (if different):

<yourPartition> <desiredMountDir> ext4 defaults 0 0

Save the file, then run the following command to mount your drive now:

mount /dev/sdb1 -v

You should now be able to navigate to your target directory and access it. If you so wish, you can also restart your Ubuntu system and check to ensure that your partition successfully auto-mounts to your desired directory.

Step 4: Configure Elasticsearch to store data within Mounted Directory

Duplicate the default Elasticsearch folder structure from /var to your mounted directory:

mkdir -p /mnt/es_data/lib/elasticsearch

For peace of mind, make the directories fully read/write accessible:

chmod -Rv 777 /mnt/es_data

Configure Elasticsearch.yml within /etc/elasticsearch to point to the mounted directory for storing data:

NOTE: This guide was performed on an Elasticsearch cluster that did NOT have any data stored yet. If your Elasticsearch cluster has been active, you will need to move any and all information from the default directory (/var/lib/elasticsearch) and copy it to your mounted directory using the same folder structure. A later guide may detail this process in full.

Step 5: Move on to your next Elasticsearch node

Repeat Steps 1-4 for all of your other Elasticsearch nodes, if any.

In a multi-node cluster you typically have three, while single-node clusters would only have the one node for all ELK services.

All set!

Your Elasticsearch cluster should be all set and configured to now store its data on a separate partition. Turn on Elasticsearch on all nodes and log in to your Kibana web interface to ensure all nodes are up, communicating, and displaying the new storage amount.

If you have any questions or comments, leave them below or contact me!

Sources

• https://linuxexpresso.wordpress.com/2010/03/14/mount-partitions-in-terminal-fstab/