Setup a 4 Node Distributed Minio Cluster

https://min.io

In this post we will setup a 4 node minio distributed cluster on AWS.

Create AWS Resources

First create the minio security group that allows port 22 and port 9000 from everywhere (you can change this to suite your needs)

Attach a secondary disk to each node, in this case I will attach a EBS disk of 20GB to each instance:

Associate the security group that was created to the instances:

After your instances has been provisioned, it will look like this:

Access the EC2 Instances

The secondary disk that we associated to our EC2 instances can be found by looking at the block devices:

$ sudo lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0     7:0    0  89M  1 loop /snap/core/7713
loop1     7:1    0  18M  1 loop /snap/amazon-ssm-agent/1480
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdb    202:16   0  20G  0 disk

The following steps will need to be applied on all 4 EC2 instances.

Switch to the root user and mount the secondary disk to the /data directory:

$ sudo su 
$ mkdir /data
$ mkfs.xfs /dev/xvdb
$ echo "/dev/xvdb /data xfs defaults,noatime,nofail 0 0" >> /etc/fstab
$ mount -a

Verify that the disk has been mounted:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.7G  1.3G  6.5G  17% /
/dev/xvdb        20G   53M   20G   1% /data

After you have mounted the disks on all 4 EC2 instances, gather the private ip addresses and set your host files on all 4 instances (in my case):

$ cat > /etc/hosts << EOF
172.31.38.98    minio-1
172.31.39.214    minio-2
172.31.37.208    minio-3
172.31.42.115    minio-4
127.0.0.1     localhost
EOF

Install Minio

Install minio on each node:

$ apt update && apt install wget -y
$ wget -O /usr/local/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod +x /usr/local/bin/minio

After minio has been installed on all the nodes, create the systemd unit files on the nodes:

$ cat > /lib/systemd/system/minio.service << EOF
[Unit]
Description=minio
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/
User=root
Group=root
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server \$MINIO_OPTS
Restart=always
LimitNOFILE=65536
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
EOF

In my case, I am setting my access key to AKaHEgQ4II0S7BjT6DjAUDA4BX and my secret key to SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH, therefore I am setting this to the minio's default configuration:

$ cat > /etc/default/minio << EOF
MINIO_OPTS="http://minio-1:9000/data http://minio-2:9000/data http://minio-3:9000/data http://minio-4:9000/data"
MINIO_ACCESS_KEY="AKaHEgQ4II0S7BjT6DjAUDA4BX"
MINIO_SECRET_KEY="SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH"
EOF

When the above step has been applied to all the nodes, reload the systemd daemon, enable the service on boot and start the service on all the nodes:

$ systemctl daemon-reload
$ systemctl enable minio
$ systemctl start minio.service

Head over to any node and run a status to see if minio has started:

$ systemctl status minio.service
  minio.service - minio
   Loaded: loaded (/lib/systemd/system/minio.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-09 17:47:15 UTC; 1min 16s ago
     Docs: https://docs.min.io
Main PID: 2089 (minio)
    Tasks: 8 (limit: 1152)
   CGroup: /system.slice/minio.service
           └─2089 /usr/local/bin/minio server http://minio-1:9000/data http://minio-2:9000/data http://minio-3:9000/data http://minio-4:9000/data


Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Status:         4 Online, 0 Offline.
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Endpoint:  http://172.31.38.98:9000  http://127.0.0.1:9000
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Browser Access:
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    http://172.31.38.98:9000  http://127.0.0.1:9000
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Object API (Amazon S3 compatible):
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    Java:       https://docs.min.io/docs/java-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    Python:     https://docs.min.io/docs/python-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]:    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide

As you can see, all 4 nodes has started.

Minio WebUI

Get the public ip of one of your nodes and access it on port 9000:

Creating your first bucket will look like this:

Using the Python API

Create a virtual environment and install minio:

$ virtualenv .venv-minio -p /usr/local/bin/python3.7  && source .venv-minio/bin/activate
$ pip install minio

Create a file that we will upload to minio:

$ echo "ok" > file.txt

Enter the python interpreter, instantiate a minio client, create a bucket and upload the text file that we created:

>>> from minio import Minio
>>> minioClient = Minio('34.248.202.30:9000', access_key='AKaHEgQ4II0S7BjT6DjAUDA4BX', secret_key='SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH', secure=False)
>>> minioClient.make_bucket('log-bucket', location='us-east-1’)
>>> minioClient.fput_object('log-bucket', 'objects/file.txt', 'file.txt’)

Let's list the objects in our newly created bucket:

>>> objects = minioClient.list_objects('log-bucket', prefix='', recursive=True)
>>> for obj in objects:
...     print(obj.object_name)
...
objects/file.txt

We can also list our buckets:

>>> buckets = minioClient.list_buckets()
>>> for bucket in buckets:
...     print(bucket.name)
...
log-bucket
my-first-bucket

For more information check out: