Today we will setup a Distributed-Replicated GlusterFS Volume.

Previous Posts:

From our GlusterFS Series we have covered the following:

What's the benefits of a Distributed-Replicated GlusterFS Volume:

Distributed Replicated Volumes is when data is distributed across replicated sets and they are great when you need data redundency and scaling.

So ideally, you can have 4 nodes in your storage cluster, consisting of 2 replicated sets, where a file is replicated between 2 nodes (having a replication count of 2), if one of the nodes goes down, you still have access to your file as the data of that particular file is replicated.

This figure will explain the above a bit better:

Setting up our Distribute-Replicated GlusterFS Volume:

In this case, I will only be using 2 nodes with 2 disk drives each, having node-1 disk-1 forming a replica set with node-2 disk-1, etc.

Installing GlusterFS:

Installing GlusterFS, dependecies and enabling GlusterFS to start on boot:

$ sudo apt update && sudo apt upgrade -y
$ sudo apt install xfsprogs attr -y
$ sudo apt install glusterfs-server glusterfs-client glusterfs-common -y
$ sudo systemctl enable glusterfs-server

The above step should be done on our 2nd node as well

Preparing the Disks:

Having a look at our block devices, we will be using xvdf and xvdg for glusterfs:

$ sudo lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk 
└─xvda1 202:1    0   8G  0 part /
xvdf    202:80   0  50G  0 disk 
xvdg    202:96   0  50G  0 disk

First we will need to format them into XFS, create our directories where we will mount these volumes to, and add them to fstab so that they can be mounted on boot time:

$ sudo mkfs.xfs /dev/xvdf
$ sudo mkfs.xfs /dev/xvdg
$ sudo mkdir -p /gluster/{a,b}
$ sudo su -c 'echo "/dev/xvdf /gluster/a xfs defaults 0 0" >> /etc/fstab'
$ sudo su -c 'echo "/dev/xvdg /gluster/b xfs defaults 0 0" >> /etc/fstab'

After we have set our disks to be mounted in fstab, we can go ahead and mount them:

$ sudo mount -a

If everything went according to plan, we should be able to view them:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
/dev/xvda1      7.7G  1.3G  6.5G  17% /
/dev/xvdf        50G   33M   50G   1% /gluster/a
/dev/xvdg        50G   33M   50G   1% /gluster/b

Create the brick directory where we will point our glusterfs brick volume:

$ sudo mkdir /gluster/{a,b}/brick

Discover the GlusterFS Nodes:

$ sudo gluster peer probe ip-172-31-47-175
peer probe: success.

List the nodes in our cluster:

$ sudo gluster pool list
UUID                    Hostname            State
44888c84-ed1c-4ec3-b3f5-a31f70ca2136    ip-172-31-47-175    Connected 
5cf398ab-077b-4184-87c3-af3b7eb650af    localhost           Connected

On our 2nd node, prepare the directories and add the entires into fstab:

$ sudo mkdir -p /gluster/{c,d}
$ sudo su -c 'echo "/dev/xvdf /gluster/c xfs defaults 0 0" >> /etc/fstab'
$ sudo su -c 'echo "/dev/xvdg /gluster/d xfs defaults 0 0" >> /etc/fstab'

Mount them and have a look at the disk layout:

$ sudo mount -a
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  3.1M   96M   4% /run
/dev/xvda1      7.7G  1.3G  6.5G  17% /
/dev/xvdf        50G   33M   50G   1% /gluster/c
/dev/xvdg        50G   33M   50G   1% /gluster/d

Now that the disks are mounted, create the brick directories:

$ sudo mkdir /gluster/{c,d}/brick

Create the GlusterFS Volume:

Back to our 1st node, let's create the Distribute-Replicated GlusterFS Volume gfs with the replication count of 2.

As mentioned before, one of the optimal setups, is having a 4 node cluster with the replication count of 2, but as you can see node 1 will have a replica set of:

  • Node1/Brick-a - Node2/Brick-b
  • Node2/Brick-c - Node2/Brick-d

Creating the volume:

$ sudo gluster volume create gfs replica 2 transport tcp \
  ip-172-31-44-169:/gluster/a/brick \
  ip-172-31-47-175:/gluster/c/brick \
  ip-172-31-44-169:/gluster/b/brick \
  ip-172-31-47-175:/gluster/d/brick

volume create: gfs: success: please start the volume to access data

After the volume has been created, let's go ahead and start the volume:

$ sudo gluster volume start gfs
volume start: gfs: success

Viewing Information about the Volume:

$ sudo gluster volume info
 
Volume Name: gfs
Type: Distributed-Replicate
Volume ID: 4b0d3931-73be-4dff-b1a5-56d791fccaea
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: ip-172-31-44-169:/gluster/a/brick
Brick2: ip-172-31-47-175:/gluster/c/brick
Brick3: ip-172-31-44-169:/gluster/b/brick
Brick4: ip-172-31-47-175:/gluster/d/brick
Options Reconfigured:
performance.readdir-ahead: on

And the Status, which will give us a layout of our volume:

$ sudo gluster volume status gfs
Status of volume: gfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick ip-172-31-44-169:/gluster/a/brick     49152     0          Y       7614 
Brick ip-172-31-47-175:/gluster/c/brick     49152     0          Y       7428 
Brick ip-172-31-44-169:/gluster/b/brick     49153     0          Y       7632 
Brick ip-172-31-47-175:/gluster/d/brick     49153     0          Y       7446 
NFS Server on localhost                     2049      0          Y       7653 
Self-heal Daemon on localhost               N/A       N/A        Y       7658 
NFS Server on ip-172-31-47-175              2049      0          Y       7467 
Self-heal Daemon on ip-172-31-47-175        N/A       N/A        Y       7472 
 
Task Status of Volume gfs
------------------------------------------------------------------------------
There are no active volume tasks

If you would like to disable NFS Support:

$ sudo gluster volume set gfs nfs.disable on

On each node, set the GlusterFS Volume to fstab so that the GlusterFS Volume can be mounted on boot time:

$ sudo su -c 'echo "localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=$(hostname) 0 0" >> /etc/fstab'

Mount the volume and if everything is going according to plan, you should be able to see your volume in the disk output:

$ sudo mount -a

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
/dev/xvda1      7.7G  1.3G  6.5G  17% /
/dev/xvdf        50G   33M   50G   1% /gluster/a
/dev/xvdg        50G   33M   50G   1% /gluster/b
localhost:/gfs  100G   65M  100G   1% /mnt

Testing the Data Replication:

Let's test the data replication, when we copy data to our volume.

On node1, write a file to our GlusterFS mounted volume, then search for the filename in the GlusterFS Brick directory:

$ echo $RANDOM > /mnt/file01.txt
$ sudo find /gluster/ -name file01.txt
/gluster/b/brick/file01.txt

You will find that the copy is in the 2nd disk's mounted directory - brick-b, which is on the 1st node, so then the other copy should be on brick-d on the 2nd node:

$ sudo find /gluster/ -name file01.txt
/gluster/d/brick/file01.txt

Let's write another file to our volume, and see what happens this time:

$ echo $RANDOM > /mnt/file02.txt
$ sudo find /gluster/ -name file02.txt
/gluster/a/brick/file02.txt

We can now see that the file was sent to the 1st disk of our 1st node on brick-a, so then the other copy should be on our 2nd node's brick-c:

$ sudo find /gluster/ -name file02.txt
/gluster/c/brick/file02.txt

Resources:

Thanks

Thanks for reading :)