Setup a GlusterFS Distributed Replicated Volume
Today we will setup a Distributed-Replicated GlusterFS Volume.
Previous Posts:
From our GlusterFS Series we have covered the following:
- GlusterFS: Distributed Replicated Volume
- GlusterFS: Distributed Storage Volume
- GlusterFS: Replicated Storage Volume
- GlusterFS: Adding Bricks to your Volume
- GlusterFS: Replace Faulty Bricks
What's the benefits of a Distributed-Replicated GlusterFS Volume:
Distributed Replicated Volumes is when data is distributed across replicated sets and they are great when you need data redundency and scaling.
So ideally, you can have 4 nodes in your storage cluster, consisting of 2 replicated sets, where a file is replicated between 2 nodes (having a replication count of 2), if one of the nodes goes down, you still have access to your file as the data of that particular file is replicated.
This figure will explain the above a bit better:
Setting up our Distribute-Replicated GlusterFS Volume:
In this case, I will only be using 2 nodes with 2 disk drives each, having node-1 disk-1 forming a replica set with node-2 disk-1, etc.
Installing GlusterFS:
Installing GlusterFS, dependecies and enabling GlusterFS to start on boot:
$ sudo apt update && sudo apt upgrade -y
$ sudo apt install xfsprogs attr -y
$ sudo apt install glusterfs-server glusterfs-client glusterfs-common -y
$ sudo systemctl enable glusterfs-server
The above step should be done on our 2nd node as well
Preparing the Disks:
Having a look at our block devices, we will be using xvdf and xvdg for glusterfs:
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 50G 0 disk
xvdg 202:96 0 50G 0 disk
First we will need to format them into XFS, create our directories where we will mount these volumes to, and add them to fstab so that they can be mounted on boot time:
$ sudo mkfs.xfs /dev/xvdf
$ sudo mkfs.xfs /dev/xvdg
$ sudo mkdir -p /gluster/{a,b}
$ sudo su -c 'echo "/dev/xvdf /gluster/a xfs defaults 0 0" >> /etc/fstab'
$ sudo su -c 'echo "/dev/xvdg /gluster/b xfs defaults 0 0" >> /etc/fstab'
After we have set our disks to be mounted in fstab, we can go ahead and mount them:
$ sudo mount -a
If everything went according to plan, we should be able to view them:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
/dev/xvda1 7.7G 1.3G 6.5G 17% /
/dev/xvdf 50G 33M 50G 1% /gluster/a
/dev/xvdg 50G 33M 50G 1% /gluster/b
Create the brick directory where we will point our glusterfs brick volume:
$ sudo mkdir /gluster/{a,b}/brick
Discover the GlusterFS Nodes:
$ sudo gluster peer probe ip-172-31-47-175
peer probe: success.
List the nodes in our cluster:
$ sudo gluster pool list
UUID Hostname State
44888c84-ed1c-4ec3-b3f5-a31f70ca2136 ip-172-31-47-175 Connected
5cf398ab-077b-4184-87c3-af3b7eb650af localhost Connected
On our 2nd node, prepare the directories and add the entires into fstab:
$ sudo mkdir -p /gluster/{c,d}
$ sudo su -c 'echo "/dev/xvdf /gluster/c xfs defaults 0 0" >> /etc/fstab'
$ sudo su -c 'echo "/dev/xvdg /gluster/d xfs defaults 0 0" >> /etc/fstab'
Mount them and have a look at the disk layout:
$ sudo mount -a
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 3.1M 96M 4% /run
/dev/xvda1 7.7G 1.3G 6.5G 17% /
/dev/xvdf 50G 33M 50G 1% /gluster/c
/dev/xvdg 50G 33M 50G 1% /gluster/d
Now that the disks are mounted, create the brick directories:
$ sudo mkdir /gluster/{c,d}/brick
Create the GlusterFS Volume:
Back to our 1st node, let's create the Distribute-Replicated GlusterFS Volume gfs with the replication count of 2.
As mentioned before, one of the optimal setups, is having a 4 node cluster with the replication count of 2, but as you can see node 1 will have a replica set of:
- Node1/Brick-a - Node2/Brick-b
- Node2/Brick-c - Node2/Brick-d
Creating the volume:
$ sudo gluster volume create gfs replica 2 transport tcp \
ip-172-31-44-169:/gluster/a/brick \
ip-172-31-47-175:/gluster/c/brick \
ip-172-31-44-169:/gluster/b/brick \
ip-172-31-47-175:/gluster/d/brick
volume create: gfs: success: please start the volume to access data
After the volume has been created, let's go ahead and start the volume:
$ sudo gluster volume start gfs
volume start: gfs: success
Viewing Information about the Volume:
$ sudo gluster volume info
Volume Name: gfs
Type: Distributed-Replicate
Volume ID: 4b0d3931-73be-4dff-b1a5-56d791fccaea
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: ip-172-31-44-169:/gluster/a/brick
Brick2: ip-172-31-47-175:/gluster/c/brick
Brick3: ip-172-31-44-169:/gluster/b/brick
Brick4: ip-172-31-47-175:/gluster/d/brick
Options Reconfigured:
performance.readdir-ahead: on
And the Status, which will give us a layout of our volume:
$ sudo gluster volume status gfs
Status of volume: gfs
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ip-172-31-44-169:/gluster/a/brick 49152 0 Y 7614
Brick ip-172-31-47-175:/gluster/c/brick 49152 0 Y 7428
Brick ip-172-31-44-169:/gluster/b/brick 49153 0 Y 7632
Brick ip-172-31-47-175:/gluster/d/brick 49153 0 Y 7446
NFS Server on localhost 2049 0 Y 7653
Self-heal Daemon on localhost N/A N/A Y 7658
NFS Server on ip-172-31-47-175 2049 0 Y 7467
Self-heal Daemon on ip-172-31-47-175 N/A N/A Y 7472
Task Status of Volume gfs
------------------------------------------------------------------------------
There are no active volume tasks
If you would like to disable NFS Support:
$ sudo gluster volume set gfs nfs.disable on
On each node, set the GlusterFS Volume to fstab so that the GlusterFS Volume can be mounted on boot time:
$ sudo su -c 'echo "localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=$(hostname) 0 0" >> /etc/fstab'
Mount the volume and if everything is going according to plan, you should be able to see your volume in the disk output:
$ sudo mount -a
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
/dev/xvda1 7.7G 1.3G 6.5G 17% /
/dev/xvdf 50G 33M 50G 1% /gluster/a
/dev/xvdg 50G 33M 50G 1% /gluster/b
localhost:/gfs 100G 65M 100G 1% /mnt
Testing the Data Replication:
Let's test the data replication, when we copy data to our volume.
On node1, write a file to our GlusterFS mounted volume, then search for the filename in the GlusterFS Brick directory:
$ echo $RANDOM > /mnt/file01.txt
$ sudo find /gluster/ -name file01.txt
/gluster/b/brick/file01.txt
You will find that the copy is in the 2nd disk's mounted directory - brick-b, which is on the 1st node, so then the other copy should be on brick-d on the 2nd node:
$ sudo find /gluster/ -name file01.txt
/gluster/d/brick/file01.txt
Let's write another file to our volume, and see what happens this time:
$ echo $RANDOM > /mnt/file02.txt
$ sudo find /gluster/ -name file02.txt
/gluster/a/brick/file02.txt
We can now see that the file was sent to the 1st disk of our 1st node on brick-a, so then the other copy should be on our 2nd node's brick-c:
$ sudo find /gluster/ -name file02.txt
/gluster/c/brick/file02.txt
Resources:
Thanks
Thanks for reading :)