3 Node Multi-master k3s Kubernetes Cluster with MySQL

In this post we will setup a 3 node multi-master kubernetes cluster using k3s which is backed with mysql.

Our Setup

Our IP Addresses for our nodes will be:

• MySQL: 192.168.0.200
• Rpi-01: 192.168.0.21
• Rpi-02: 192.168.0.22
• Rpi-03: 192.168.0.23

Database

On our database node, i'm installing mariadb server 10.3:

$ sudo apt install mariadb-server-10.3 -y

The initial setup of our database for kubernetes:

$ sudo mysql_secure_password

Then enable the plugin for socket authentication:

$ sudo mysql -u root
> use mysql;
> UPDATE user SET plugin='' WHERE User='root';
> FLUSH PRIVILEGES;
> exit;

Now you can auth with your password:

$ mysql -u root -p

Create the k3s database, the rancher user that will own the database and set a password:

> CREATE database k3s;
> CREATE user 'rancher'@'%' IDENTIFIED BY 'password';
> GRANT ALL on k3s.* TO 'rancher'@'%';
> FLUSH PRIVILEGES'

> use mysql;
> UPDATE user SET plugin='' WHERE User='rancher';
> FLUSH PRIVILEGES;

Edit sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf and comment out bind-address so that mysql listens on all interfaces, then restart:

$ sudo /etc/init.d/mysql restart

our database details for k3s:

host: 192.168.0.200
user: rancher
pass: password
db: k3s

Setup Kubernetes Nodes

On our first node, install k3s and use the arguments as you prefer, but the important one is to point our database using --datastore-endpoint .

You will notice that I am also using --node-taint k3s-controlplane=true:NoExecute" on all 3 nodes, you can remove that, but I also want to show how you can remove the taints manually from node2 and node3 after the deployment.

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --token secret-01010101-xabcdefx --tls-san k3s.mydomain.com --node-taint k3s-controlplane=true:NoExecute" sh -s - server --datastore-endpoint="mysql://rancher:password@tcp(192.168.0.200:3306)/k3s"

Your kubeconfig will be available at: /etc/rancher/k3s/k3s.yaml

When we list our kubernetes nodes:

$ kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
rpi-01   Ready    master   16s   v1.18.4+k3s1

Do the same for node 2 and node 3, once they have been provisioned, you should see the following:

$ kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
rpi-01   Ready    master   2m30s   v1.18.4+k3s1
rpi-02   Ready    master   41s     v1.18.4+k3s1
rpi-03   Ready    master   25s     v1.18.4+k3s1

Untaint Nodes

As mentioned, all 3 nodes run with NoExecute taints, and now we would like to remove them from node2 and node3 as a demonstration, if you would like to do that.

Describe the nodes to view the taints:

$ kubectl describe nodes | grep Taints
Taints:             k3s-controlplane=true:NoExecute
Taints:             k3s-controlplane=true:NoExecute
Taints:             k3s-controlplane=true:NoExecute

Remove taint from node2:

$ kubectl taint nodes rpi-02 k3s-controlplane=true:NoExecute-
node/rpi-02 untainted

And remove from node3:

$ kubectl taint nodes rpi-03 k3s-controlplane=true:NoExecute-
node/rpi-03 untainted

Now when we describe our nodes again:

$ kubectl describe nodes | grep Taints
Taints:             k3s-controlplane=true:NoExecute
Taints:             <none>
Taints:             <none>

Deploy a sample application

To test our cluster, let's deploy a sample web application, which is a basic application that returns it's hostname:

$ wget https://gist.githubusercontent.com/ruanbekker/d3efa0ff72a6d26698e6fb3a0839886d/raw/2bba233ec7a06a866dee9aa9daa106960f3a4369/deployment-green-hostname-rpi.yaml 

$ kubectl apply -f deployment-green-hostname-rpi.yaml
namespace/color created
deployment.apps/green created
service/green created
ingress.extensions/colors-green created

Once we deployed our application, have a look at the deployment and pods:

$ kubectl get deployment,pod -n color
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/green   4/4     4            4           2m1s

NAME                         READY   STATUS    RESTARTS   AGE
pod/green-545d49b94b-cxt84   1/1     Running   0          118s
pod/green-545d49b94b-84fdm   1/1     Running   0          117s
pod/green-545d49b94b-gghnf   1/1     Running   0          118s
pod/green-545d49b94b-d5rvf   1/1     Running   0          117s

And our ingress:

$ kubectl get ingress -n color
NAME           CLASS    HOSTS                        ADDRESS         PORTS   AGE
colors-green   <none>   hostname-green.homedns.xyz   192.168.0.21    80      2m49s

Make a http request to our application:

$ curl -H "Host:hostname-green.homedns.xyz" http://192.168.0.21
Green running on: green-545d49b94b-84fdm

Thank You

Thanks for reading, hope you found it useful.