Setup a 4 Node Distributed Minio Cluster
In this post we will setup a 4 node minio distributed cluster on AWS.
Create AWS Resources
First create the minio security group that allows port 22 and port 9000 from everywhere (you can change this to suite your needs)
Attach a secondary disk to each node, in this case I will attach a EBS disk of 20GB to each instance:
Associate the security group that was created to the instances:
After your instances has been provisioned, it will look like this:
Access the EC2 Instances
The secondary disk that we associated to our EC2 instances can be found by looking at the block devices:
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 89M 1 loop /snap/core/7713
loop1 7:1 0 18M 1 loop /snap/amazon-ssm-agent/1480
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 20G 0 disk
The following steps will need to be applied on all 4 EC2 instances.
Switch to the root user and mount the secondary disk to the /data directory:
$ sudo su
$ mkdir /data
$ mkfs.xfs /dev/xvdb
$ echo "/dev/xvdb /data xfs defaults,noatime,nofail 0 0" >> /etc/fstab
$ mount -a
Verify that the disk has been mounted:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.7G 1.3G 6.5G 17% /
/dev/xvdb 20G 53M 20G 1% /data
After you have mounted the disks on all 4 EC2 instances, gather the private ip addresses and set your host files on all 4 instances (in my case):
$ cat > /etc/hosts << EOF
172.31.38.98 minio-1
172.31.39.214 minio-2
172.31.37.208 minio-3
172.31.42.115 minio-4
127.0.0.1 localhost
EOF
Install Minio
Install minio on each node:
$ apt update && apt install wget -y
$ wget -O /usr/local/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod +x /usr/local/bin/minio
After minio has been installed on all the nodes, create the systemd unit files on the nodes:
$ cat > /lib/systemd/system/minio.service << EOF
[Unit]
Description=minio
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local/
User=root
Group=root
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server \$MINIO_OPTS
Restart=always
LimitNOFILE=65536
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
EOF
In my case, I am setting my access key to AKaHEgQ4II0S7BjT6DjAUDA4BX and my secret key to SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH, therefore I am setting this to the minio's default configuration:
$ cat > /etc/default/minio << EOF
MINIO_OPTS="http://minio-1:9000/data http://minio-2:9000/data http://minio-3:9000/data http://minio-4:9000/data"
MINIO_ACCESS_KEY="AKaHEgQ4II0S7BjT6DjAUDA4BX"
MINIO_SECRET_KEY="SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH"
EOF
When the above step has been applied to all the nodes, reload the systemd daemon, enable the service on boot and start the service on all the nodes:
$ systemctl daemon-reload
$ systemctl enable minio
$ systemctl start minio.service
Head over to any node and run a status to see if minio has started:
$ systemctl status minio.service
minio.service - minio
Loaded: loaded (/lib/systemd/system/minio.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-10-09 17:47:15 UTC; 1min 16s ago
Docs: https://docs.min.io
Main PID: 2089 (minio)
Tasks: 8 (limit: 1152)
CGroup: /system.slice/minio.service
└─2089 /usr/local/bin/minio server http://minio-1:9000/data http://minio-2:9000/data http://minio-3:9000/data http://minio-4:9000/data
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Status: 4 Online, 0 Offline.
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Endpoint: http://172.31.38.98:9000 http://127.0.0.1:9000
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Browser Access:
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: http://172.31.38.98:9000 http://127.0.0.1:9000
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Object API (Amazon S3 compatible):
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Go: https://docs.min.io/docs/golang-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Java: https://docs.min.io/docs/java-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: Python: https://docs.min.io/docs/python-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Oct 09 17:47:24 ip-172-31-38-98 minio[2089]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
As you can see, all 4 nodes has started.
Minio WebUI
Get the public ip of one of your nodes and access it on port 9000:
Creating your first bucket will look like this:
Using the Python API
Create a virtual environment and install minio:
$ virtualenv .venv-minio -p /usr/local/bin/python3.7 && source .venv-minio/bin/activate
$ pip install minio
Create a file that we will upload to minio:
$ echo "ok" > file.txt
Enter the python interpreter, instantiate a minio client, create a bucket and upload the text file that we created:
>>> from minio import Minio
>>> minioClient = Minio('34.248.202.30:9000', access_key='AKaHEgQ4II0S7BjT6DjAUDA4BX', secret_key='SKFzHq5iDoQgF7gyPYRFhzNMYSvY6ZFMpH', secure=False)
>>> minioClient.make_bucket('log-bucket', location='us-east-1’)
>>> minioClient.fput_object('log-bucket', 'objects/file.txt', 'file.txt’)
Let's list the objects in our newly created bucket:
>>> objects = minioClient.list_objects('log-bucket', prefix='', recursive=True)
>>> for obj in objects:
... print(obj.object_name)
...
objects/file.txt
We can also list our buckets:
>>> buckets = minioClient.list_buckets()
>>> for bucket in buckets:
... print(bucket.name)
...
log-bucket
my-first-bucket
For more information check out: