/ Docker

Docker Swarm: Getting Started with a 3 Node Docker Swarm Cluster with a Scalable App

What are we up to today?

  • Install Docker and Docker Compose on 3 Nodes
  • Initialize the Swarm and Join the Worker Nodes
  • Create a Nginx Service with 2 Replicas
  • Do some Inspection: View some info on the Service
  • Demo: Create a Registry, Flask App with Python that Reports the Hostname, Scale the App, Change the Code, Update the Service, etc.

For detailed information on Docker Swarm, have a look at their Docker's Documentation.

Getting Started:

Bootstrap Docker Swarm Setup with Docker Compose on Ubuntu 16.04:

Installing Docker:

Run the following on all 3 Nodes as the root user:

$ apt-get update && apt-get upgrade -y
$ apt-get remove docker docker-engine -y
$ apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y

$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo $ apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"

$ apt-get update
$ apt-get install docker-ce -y
$ systemctl enable docker
$ systemctl restart docker

$ curl -L https://github.com/docker/compose/releases/download/1.13.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose
$ usermod -aG docker <your-user>

Testing:

Test Docker by running a Test Container

$ docker --version
$ docker-compose --version
$ docker run hello-world

In this example I do not rely on DNS, and for simplicity, I just added the host entry data into the /etc/hosts files on each node:

$ cat /etc/hosts
172.31.18.90    manager
172.31.20.94    worker-1
172.31.21.50    worker-2

Initialize the Swarm:

Now we will initialize the swarm on the manager node and as we have more than one network interface, we will specify the --advertise-addr option:

[manager] $ docker swarm init --advertise-addr 172.31.18.90
Swarm initialized: current node (siqyf3yricsvjkzvej00a9b8h) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 \
    172.31.18.90:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

If this is a scenario where we would like to add more than one manager, we would need to run the above command to add more managers.

Join the Worker nodes to the Manager:

Now, to join the worker nodes to the swarm, we will run the docker swarm join command that we received in the swarm initialisation step, like below:

[worker-1] $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 172.31.18.90:2377
This node joined a swarm as a worker.

And to join the second worker to the swarm:

[worker-2] $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 172.31.18.90:2377
This node joined a swarm as a worker.

To see the node status, so that we can determine if the nodes are active/available etc, from the manager node, list all the nodes in the swarm:

[manager] $ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
j14mte3v1jhtbm3pb2qrpgwp6    worker-1  Ready   Active
siqyf3yricsvjkzvej00a9b8h *  master    Ready   Active        Leader
srl5yzme5hxnzxal2t1efmwje    worker-2  Ready   Active

If at any time, you lost your join token, it can be retrieved by running the following for the manager token:

$ docker swarm join-token manager -q
SWMTKN-1-67chzvi4epx28ii18gizcia8idfar5hokojz660igeavnrltf0-09ijujbnnh4v960b8xel58pmj

And the following to retrieve the worker token:

$ docker swarm join-token worker -q
SWMTKN-1-67chzvi4epx28ii18gizcia8idfar5hokojz660igeavnrltf0-acs21nn28v17uwhw0oqg5ibwx

At this moment, we will see that we have no services running in our swarm:

[manager] $ docker service ls
ID  NAME  MODE  REPLICAS  IMAGE

Deploying our First Service:

Let's create a nginx service with 2 replicas, which means that there will be 2 containers of nginx running in our swarm.

If any of these containers fail, they will be spawned again to have the desired number that we set on the replica option:

[manager] $ docker service create --name my-web --publish 8080:80 --replicas 2 nginx

Let's have a look at our nginx service:

[manager] $ docker service ls
ID            NAME    MODE        REPLICAS  IMAGE
1okycpshfusq  my-web  replicated  2/2       nginx:latest

After we see that the replica count is 2/2 our service is ready.

To see on what nodes our containers are running that makes up our service:

[manager] $ docker service ps my-web
ID            NAME      IMAGE         NODE      DESIRED STATE  CURRENT STATE           ERROR  PORTS
k0qqrh8s0c2d  my-web.1  nginx:latest  worker-1  Running        Running 30 seconds ago
nku9wer6tmll  my-web.2  nginx:latest  worker-2  Running        Running 30 seconds ago

We can also retrieve more information of our service by using the inspect option:

[manager] $ docker service inspect my-web
[
    {
        "ID": "1okycpshfusqos8023gtvdf11",
        "Version": {
            "Index": 22
        },
        "CreatedAt": "2017-06-21T07:58:15.177547236Z",
        "UpdatedAt": "2017-06-21T07:58:15.178864919Z",
        "Spec": {
            "Name": "my-web",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "nginx:latest@sha256:41ad9967ea448d7c2b203c699b429abe1ed5af331cd92533900c6d77490e0268",
                    "DNSConfig": {}
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "MaxAttempts": 0
                },
                "Placement": {},
                "ForceUpdate": 0
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 2
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "MaxFailureRatio": 0
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 8080,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 8080,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 8080,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "wvim41tbf7tsmi7z49g56bisa",
                    "Addr": "10.255.0.6/16"
                }
            ]
        },
        "UpdateStatus": {
            "StartedAt": "0001-01-01T00:00:00Z",
            "CompletedAt": "0001-01-01T00:00:00Z"
        }
    }
]

We can get the Endpoint Port info by using inspect and using the --format parameter to filter the output:

[manager] $ docker service inspect --format="{{json .Endpoint.Ports}}" my-web  | python -m json.tool

From the output we will find the PublishedPort is the Port that we Expose, which will be the listener. Our TargetPort will be the port that is listening on the container:

[
    {
        "Protocol": "tcp",
        "PublishMode": "ingress",
        "PublishedPort": 8080,
        "TargetPort": 80
    }
]

To get the Virtual IP Info:

[manager] $ docker service inspect --format="{{json .Endpoint.VirtualIPs}}" my-web  | python -m json.tool
[
    {
        "Addr": "10.255.0.6/16",
        "NetworkID": "wvim41tbf7tsmi7z49g56bisa"
    }
]

To only get the Published Port:

[manager] $ docker service inspect --format="{{range .Endpoint.Ports}}{{.PublishedPort}}{{end}}" my-web
8080

We can also list our networks:

[manager] $ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
1cb24ee7e385        bridge              bridge              local
503b2f0eda49        docker_gwbridge     bridge              local
fe679d82d502        host                host                local
wvim41tbf7ts        ingress             overlay             swarm
310b9219ec0c        none                null                local

And also inspecting our network for more information:

[manager] $ docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "wvim41tbf7tsmi7z49g56bisa",
        "Created": "2017-06-21T07:50:44.14232108Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "b6b2a8f3f233cfc77806718a9c47daf02ad128ba81c96d6382ff1d3799c3b5c1",
                "MacAddress": "02:42:0a:ff:00:03",
                "IPv4Address": "10.255.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "master-59041a33bebc",
                "IP": "172.31.18.90"
            },
            {
                "Name": "worker-1-cfee817ddb5f",
                "IP": "172.31.20.94"
            },
            {
                "Name": "worker-2-40891fb1fa3f",
                "IP": "172.31.21.50"
            }
        ]
    }
]

To get the IP of container in a swarm cluster:

[manager] $ docker service ps my-web
ID            NAME      IMAGE         NODE      DESIRED STATE  CURRENT STATE           ERROR  PORTS
k0qqrh8s0c2d  my-web.1  nginx:latest  worker-1  Running        Running 31 minutes ago
nku9wer6tmll  my-web.2  nginx:latest  worker-2  Running        Running 31 minutes ago

For example, we would like to determine the IP for our my-web.1 nginx service on worker-1:

[worker-1] $ docker ps
CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
72e05af4e654

Then inspect the container to get the IP Address:

[worker-1] $ docker inspect 72e05af4e654 --format "{{json .NetworkSettings.Networks.ingress.IPAddress}}"
"10.255.0.8"

or:

[worker-1] $ docker inspect 72e05af4e654 --format "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}"
10.255.0.8

Creating the Demo Web Application Service Example:

In this example, we will create a Python Flask Example that prints out the hostname of the container (containerid) and also prints a UUID.

So when we scale the service we can see that the round-robin load balancing algorithm, reports a different hostname on each HTTP Request.

Creating a Private Registry to Save our Images:

We will deploy a private registry server as a service into our swarm that will act as a private registry that will host our images that we will be pushing to:

[manager] $ docker service create --name registry --publish "5000:5000" registry:2

Create a Network:

We will create a network, which will use the overlay driver which spans multiple docker hosts, that makes up the overlay network and specify the subnet of choice:

[manager] $ docker network create --driver overlay --subnet 10.24.90.0/24 mynet

Now that our network is created, let's inspect the network to determine the detail under the hood:

$ docker network inspect mynet
[
    {
        "Name": "mynet",
        "Id": "byzlzl4r2mwv4j9jisu7ellhs",
        "Created": "0001-01-01T00:00:00Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.24.90.0/24",
                    "Gateway": "10.24.90.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4103"
        },
        "Labels": null
    }
]

Also, when service is associated with this network, in terms of dns resolution, the service name will respond to the virtual ip (vip) of the service.

Create the Local Directory, Code, Dockerfile and docker-compose config:

Create the Directory where our Code will be Stored Locally:

$ mkdir flask-demo
$ cd flask-demo

Set the Python Dependency in the requirements.txt file:

$ cat requirements.txt
flask

Creating a Python Flask App (app.py) to Show the Container Name and Random uuid String:

$ cat app.py
from flask import Flask
import os
import uuid

app = Flask(__name__)

@app.route('/')
def index():
    hostname = os.uname()[1]
    randomid = uuid.uuid4()
    return 'Container Hostname: ' + hostname + ' , ' + 'UUID: ' + str(randomid)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5098)

Create the Dockerfile that will act as our instruction on how to build our image:

$ cat Dockerfile
FROM python:3.4-alpine
ADD . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Create the Docker Compose file, that will build our image using our Dockerfile, specifying the exposed ports, the image registry and we will also specify to use our existing mynet network:

$ cat docker-compose.yml
version: '3'

services:
  web:
    image: master:5000/flask-app
    build: .
    ports:
      - "80:5098"

networks:
  default:
    external:
      name: mynet

Testing locally on the host:

$ docker-compose up
 # do some testing
$ docker-compose down

Build the Image, the push to the private registry that is specified in the docker-compose.yml file:

$ docker-compose build
$ docker-compose push
Pushing web (master:5000/flask-app:latest)...
The push refers to a repository [master:5000/flask-app]
a8deddc4e222: Pushed
2916dd76ba80: Pushed
e03c530e5673: Layer already exists
0342f4a51cd7: Layer already exists
5e94921eba94: Layer already exists
ed06208397d5: Layer already exists
5accac14015f: Layer already exists
latest: digest: sha256:47b26870a38e3e468f44f81f4bf91ab9aca0637714f5b8795fccf2a85e29c8e1 size: 1786

Now that we have our image saved in our private registry, create the service from that image and the published port 80 that will translate to port 5098 on the containers that is set in the code.

Note that we set only 1 replica, which means that every time we make a HTTP Request, we will be returned with the same hostname as we only have 1 container in our swarm.

We also set --update-delay to 5 seconds so this means, when we update our service, each task (container) will be updated with 5 seconds delay between each one.

Creating the Flask-Demo Service:

[manager] $ docker service create --name flask-demo --network mynet --update-delay 5s --publish 80:5098 --replicas 1 master:5000/flask-app

Listing our Services in our Swarm:

List the services in our swarm and check if the replica number is the same as the desired count that was set when the service was created:

[manager] $ docker service ls
ID            NAME        MODE        REPLICAS  IMAGE
5y520y6fau5j  flask-demo  replicated  1/1       master:5000/flask-app:latest
xfk2z5s2ybcg  registry    replicated  1/1       registry:2

After we can see that the desired number is set, we can also run a ps to see the state and running time, as well as on which node the container lives:

$ docker service ps flask-demo
ID            NAME          IMAGE                         NODE      DESIRED STATE  CURRENT STATE           ERROR  PORTS
u7ab0xhwrlzh  flask-demo.1  master:5000/flask-app:latest  worker-1  Running        Running 11 seconds ago

Double checking, logging onto worker-1 and determine if the container lives on the mentioned node:

[worker-1] $ docker ps --filter "name=flask-demo" --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID        NAMES
4e6bd7a8a2bf        flask-demo.1.u7ab0xhwrlzh3r8xme08ievop

For just some verification, lets have a look at the exposed port:

$ docker service inspect --format="{{range .Endpoint.Ports}}{{.PublishedPort}}{{end}}" flask-demo
80

The awesome thing here is that docker swarm uses the ingress routing mesh, so that all the nodes that is part of the swarm will answer requests on the exposed port, even if that container does not reside on that node.

For example, we can see that our container resides on worker-1 so we will get the Public IP of our manager node and our worker node, then make HTTP Requests on port 80 against the public IP's and you will see that the requests gets served.

Testing our Service:

On our Manager Node:

$ curl ip.ruanbekker.com
52.214.150.151

On our Worker Node:

[worker-1] $ curl ip.ruanbekker.com
34.253.158.130
```<p>

Now from a client instance outside our swarm, we will make an HTTP Request to our Manager and Worker Node IP:

```bash
[client] $ curl -XGET 52.214.150.151
Container Hostname: 4e6bd7a8a2bf , UUID: 771cc6bd-b669-4f68-83c6-cbb30ad01380[client] $

[client] $ curl -XGET 34.253.158.130
Container Hostname: 4e6bd7a8a2bf , UUID: c4f4115b-e848-4941-a399-b703afed249d[client] $

We can see we need to change the code so that it ends with a new line character

Updating our Application Code:

$ vi app.py

Change the code so that we add a new line character after our UUID:

from

return 'Container Hostname: ' + hostname + ' , ' + 'UUID: ' + str(randomid)

to:

return 'Container Hostname: ' + hostname + ' , ' + 'UUID: ' + str(randomid) + '\n'

Build the Image from the Updated Code, Push the Image to our Private Registry:

Now that our app is updated, we should build the image and push to our registry. We can apply a tag to our image like flask-app:v1 but sine we don't specify a tag, it will use the latest tag.

If you use docker-compose the tag should be set via the docker-compose.yml file, and if docker-compose is not used, you should set the tag via docker build option:

$ docker-compose build
$ docker-compose push
Pushing web (master:5000/flask-app:latest)...
The push refers to a repository [master:5000/flask-app]

Optionally, we can also build an image without docker-compose:

$ docker build -t flask-app .
$ docker tag flask-app master:5000/flask-app
$ docker push master:5000/flask-app
The push refers to a repository [master:5000/flask-app]

Lets make a Request to our API on our Private Registry to make sure we can see our repository:

$ curl master:5000/v2/_catalog
{"repositories":["flask-app"]}

Verify, if we can see the updated Image from our Private Repository:

$ docker images master:5000/flask-app
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
master:5000/flask-app   latest              3253d71edfa4        17 minutes ago      93.3 MB
master:5000/flask-app   <none>              b7fd5b5e7892        51 minutes ago      93.3 MB

Update the Service:

Apply a rolling update to a service so that our application can be updated:

$ docker service update --image master:5000/flask-app:latest flask-demo

Make sure the service is in service:

$ docker service ls
ID            NAME        MODE        REPLICAS  IMAGE
5y520y6fau5j  flask-demo  replicated  1/1       master:5000/flask-app:latest
xfk2z5s2ybcg  registry    replicated  1/1       registry:2

If we run a docker service ps on our service name, we should see that the previous container was shut down, and a new container with the updated code has been spawned:

$ docker service ps flask-demo
ID            NAME              IMAGE                         NODE      DESIRED STATE  CURRENT STATE           ERROR  PORTS
ltvenwz4jidh  flask-demo.1      master:5000/flask-app:latest  worker-2  Running        Running 2 minutes ago
u7ab0xhwrlzh   \_ flask-demo.1  master:5000/flask-app:latest  worker-1  Shutdown       Shutdown 2 minutes ago

(Optionally:) Once again, you could logon to worker-2 to make sure we can see the container-id:

[worker-2] $ docker ps --filter "name=flask-demo" --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID        NAMES
499b2a76da67        flask-demo.1.ltvenwz4jidhj7esc89ee6gzp

Testing our Updated Web Application:

Now let's make those same HTTP Requests, to see if the service was updated with the latest image:

[client] $ curl -XGET 52.214.150.151
Container Hostname: 499b2a76da67 , UUID: b1d534eb-e737-46ce-91e2-5800bf24a72e

[client] $ curl -XGET 52.214.150.151
Container Hostname: 499b2a76da67 , UUID: 6c47b0c5-7ffa-4501-a522-c430e420b12c

We can see its working, as the task got updated, running on a new container.

Scaling our Application:

Lets scale our application to 10 containers:

$ docker service scale flask-demo=10
flask-demo scaled to 10

Let's have a look at our replica count:

$ docker service ls
ID            NAME        MODE        REPLICAS  IMAGE
5y520y6fau5j  flask-demo  replicated  10/10     master:5000/flask-app:latest
xfk2z5s2ybcg  registry    replicated  1/1       registry:2

We can see our tasks is replicated and we could further see on which node they are running:

$ docker service ps flask-demo
ID            NAME              IMAGE                         NODE      DESIRED STATE  CURRENT STATE           ERROR  PORTS
ltvenwz4jidh  flask-demo.1      master:5000/flask-app:latest  worker-2  Running        Running 2 minutes ago
xyxi33keqdny   \_ flask-demo.1  master:5000/flask-app:latest  worker-1  Shutdown       Shutdown 2 minutes ago
u7ab0xhwrlzh   \_ flask-demo.1  master:5000/flask-app:latest  worker-1  Shutdown       Shutdown 6 minutes ago
yhnf8wyrimvb  flask-demo.2      master:5000/flask-app:latest  master    Running        Running 39 seconds ago
vxghtq86537m  flask-demo.3      master:5000/flask-app:latest  worker-1  Running        Running 37 seconds ago
f3p8ym23r4ir  flask-demo.4      master:5000/flask-app:latest  worker-1  Running        Running 37 seconds ago
q9hgb8z58ybd  flask-demo.5      master:5000/flask-app:latest  master    Running        Running 39 seconds ago
qdxef5d2bspt  flask-demo.6      master:5000/flask-app:latest  worker-1  Running        Running 37 seconds ago
a7wjn1lo9jbz  flask-demo.7      master:5000/flask-app:latest  worker-2  Running        Running 41 seconds ago
776oi10gyo32  flask-demo.8      master:5000/flask-app:latest  worker-1  Running        Running 37 seconds ago
u2mpr2t7aoj5  flask-demo.9      master:5000/flask-app:latest  master    Running        Running 39 seconds ago
kypazgcch5aa  flask-demo.10     master:5000/flask-app:latest  worker-2  Running        Running 41 seconds ago

More verification, viewing the containers per node where the containers are running on:

[master] $ docker ps --filter "name=flask-demo" --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID        NAMES
9e6b28ba711f        flask-demo.9.u2mpr2t7aoj5c5nggr2k3dvjf
4c3edef5c69b        flask-demo.2.yhnf8wyrimvbg937dt6tkjwb0
5a052e37ef73        flask-demo.5.q9hgb8z58ybd07cztmiqnqogy

And on the worker-1 node:

[worker-1] $ docker ps --filter "name=flask-demo" --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID        NAMES
499b2a76da67        flask-demo.6.qdxef5d2bspt2dtga4pdj6ab1
3bd92ade8242        flask-demo.4.f3p8ym23r4iredo7xqyt1yt7h
bd64090dc3f5        flask-demo.3.vxghtq86537mpxpo5t31zm1i4
3a5effab8827        flask-demo.8.776oi10gyo32h76czx7jqwkgf

Further, on the worker-2 node:

[worker-2] $ docker ps --filter "name=flask-demo" --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID        NAMES
20e8148ab72f        flask-demo.10.kypazgcch5aaw4l04ha8r4elv
217c86d234de        flask-demo.7.a7wjn1lo9jbzv4p5hbgierp7g
a186ed70646c        flask-demo.1.ltvenwz4jidhj7esc89ee6gzp

As the service is replicated to multiple containers, our application should now serve the requests from different containers, as we can see is expected from the below output:

[client] $ curl -XGET 52.214.150.151
Container Hostname: 9e6b28ba711f , UUID: 16780535-8c5f-445a-a1dd-fdb10e07bc59

[client] $ curl -XGET 52.214.150.151
Container Hostname: 4c3edef5c69b , UUID: ea496a8f-b49d-470b-a0f4-e695810374aa

Let's make 10 requests, to see the hostname reporting from the containers:

$ for x in {1..10}; do curl -XGET 52.214.150.151; done

Container Hostname: 3a5effab8827 , UUID: 00e3454e-4d0c-48d9-96ec-e8875c893b36
Container Hostname: 9e6b28ba711f , UUID: bf6da863-94a3-445d-b386-25e290004ed2
Container Hostname: 4c3edef5c69b , UUID: 1a611f14-0cc6-4495-b246-5bb4db65a002
Container Hostname: 5a052e37ef73 , UUID: 684f00dc-a17c-418a-bfb7-d2bac50d4c6e
Container Hostname: 20e8148ab72f , UUID: 85696153-d3d5-4f7a-b663-eb4e0c199555
Container Hostname: 217c86d234de , UUID: df7d2bd8-4cae-4150-a825-754e84709724
Container Hostname: a186ed70646c , UUID: 88cc84fe-44f7-4d4d-9f67-9479d52688d7
Container Hostname: 499b2a76da67 , UUID: e4d643e0-10e6-452f-abc7-f712fdf07cdd
Container Hostname: 3bd92ade8242 , UUID: ce23f3b2-c55d-4533-b164-5f60b8bf3aa0
Container Hostname: bd64090dc3f5 , UUID: 9e2bb8c2-848e-49ff-99d8-dff47644b74b

Let's make 12 requests, so in theory, we should see 10 unique hostnames:

$ for x in {1..12}; do curl -XGET 52.214.150.151; done

Container Hostname: 3a5effab8827 , UUID: 84a874b9-3767-4b07-82da-ba2ede0e2240
Container Hostname: 9e6b28ba711f , UUID: 0e1a7736-e89b-4c30-9739-97ef7b8a203a
Container Hostname: 4c3edef5c69b , UUID: 455df8e9-b465-4b58-865e-56d708148c26
Container Hostname: 5a052e37ef73 , UUID: 7478f449-ec48-4378-8cd1-fa0ae3465f78
Container Hostname: 20e8148ab72f , UUID: 25dd11fa-c09d-437a-b4d8-a12b7933a245
Container Hostname: 217c86d234de , UUID: b4e26634-037a-4694-b849-896c0bc89946
Container Hostname: a186ed70646c , UUID: 3af5619d-bfa9-4ac7-8190-0f11876b33b1
Container Hostname: 499b2a76da67 , UUID: e38bd20d-c9e7-4b1e-9f1f-a25574207558
Container Hostname: 3bd92ade8242 , UUID: 3caec655-a4f8-4956-a5b0-00a68db99233
Container Hostname: bd64090dc3f5 , UUID: 2c8cfe74-42ff-4463-b069-03fb3be5f290
Container Hostname: 3a5effab8827 , UUID: bb84adea-1dee-4b57-aea5-e086e0a979ce
Container Hostname: 9e6b28ba711f , UUID: 72706b15-544b-4a3f-b66c-5e1d3a363cc7

Some bash scripting to count any duplicates:

$ for x in {1..10}; do curl -XGET 52.214.150.151; done >> run1.txt

$ for x in {1..12}; do curl -XGET 52.214.150.151; done >> run2.txt

Our first run (10 Requests):

[client] $ cat run1.txt | awk '{print $3}' | sort | uniq -c
      1 20e8148ab72f
      1 217c86d234de
      1 3a5effab8827
      1 3bd92ade8242
      1 499b2a76da67
      1 4c3edef5c69b
      1 5a052e37ef73
      1 9e6b28ba711f
      1 a186ed70646c
      1 bd64090dc3f5

Our second run (12 Requests):

[client] $ cat run2.txt | awk '{print $3}' | sort | uniq -c
      1 20e8148ab72f
      1 217c86d234de
      2 3a5effab8827
      1 3bd92ade8242
      1 499b2a76da67
      1 4c3edef5c69b
      1 5a052e37ef73
      2 9e6b28ba711f
      1 a186ed70646c
      1 bd64090dc3f5

Python Script to Randomly select one of the 3 docker hosts to do the HTTP GET Request, and return the Docker Host IP and the Container ID that was serving the response:

>>> import time
>>> import random
>>> import requests
>>> for x in xrange(20):
...     time.sleep(0.125)
...     docker_host = random.choice(['34.253.158.130', '34.251.150.142', '52.214.150.151'])
...     get_request = requests.get('http://' + str(docker_host)).content.split(' ')[2]
...     print("Host: {0}, Container: {1}".format(docker_host, get_request))
...
Host: 52.214.150.151, Container: 217c86d234de
Host: 52.214.150.151, Container: a186ed70646c
Host: 34.251.150.142, Container: bd64090dc3f5
Host: 34.251.150.142, Container: 3a5effab8827
Host: 52.214.150.151, Container: 499b2a76da67
Host: 52.214.150.151, Container: 3bd92ade8242
Host: 52.214.150.151, Container: bd64090dc3f5
Host: 52.214.150.151, Container: 3a5effab8827
Host: 34.253.158.130, Container: 9e6b28ba711f
Host: 52.214.150.151, Container: 9e6b28ba711f
Host: 52.214.150.151, Container: 4c3edef5c69b
Host: 34.251.150.142, Container: 4c3edef5c69b
Host: 34.251.150.142, Container: 9e6b28ba711f
Host: 34.253.158.130, Container: 4c3edef5c69b
Host: 34.253.158.130, Container: 5a052e37ef73
Host: 34.253.158.130, Container: 20e8148ab72f
Host: 34.251.150.142, Container: 5a052e37ef73
Host: 34.253.158.130, Container: 217c86d234de
Host: 52.214.150.151, Container: 5a052e37ef73
Host: 34.251.150.142, Container: 217c86d234de

Some Issues I Ran Into:

"server gave HTTP response to HTTPS client" issue:
- https://github.com/docker/distribution/issues/1874

To fix this issue, on each node create a /etc/docker/daemon.json file and populate the file with the following info:

cat /etc/docker/daemon.json
{ "insecure-registries":["master:5000"] }

Manager Scheduling Only:

We can set our manager to only serve requests and do scheduling, by draining the node, all containers will be moved from this node to the other nodes in the swarm:

$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
j14mte3v1jhtbm3pb2qrpgwp6    worker-1  Ready   Active
siqyf3yricsvjkzvej00a9b8h *  master    Ready   Active        Leader
srl5yzme5hxnzxal2t1efmwje    worker-2  Ready   Active

After viewing our nodes in the swarm, we can select which node we want to drain

$ docker node update --availability drain manager
manager

To verify our change:

$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
j14mte3v1jhtbm3pb2qrpgwp6    worker-1  Ready   Active
siqyf3yricsvjkzvej00a9b8h *  master    Ready   Drain         Leader
srl5yzme5hxnzxal2t1efmwje    worker-2  Ready   Active

Inspecting our Manager Node:

$ docker node inspect --pretty manager
ID:                     siqyf3yricsvjkzvej00a9b8h
Hostname:               master
Joined at:              2017-06-21 07:50:43.525020216 +0000 utc
Status:
 State:                 Ready
 Availability:          Drain
 Address:               172.31.18.90
Manager Status:
 Address:               172.31.18.90:2377
 Raft Status:           Reachable
 Leader:                Yes
Platform:
 Operating System:      linux
 Architecture:          x86_64
Resources:
 CPUs:                  1
 Memory:                3.673 GiB
Plugins:
  Network:              bridge, host, macvlan, null, overlay
  Volume:               local
Engine Version:         17.03.1-ce

And now we can see that our containers is only running on the worker nodes:

$ docker service ps flask-demo | grep Runn
lig9wj70c5m8  flask-demo.1      master:5000/newapp:v1      worker-1  Running        Running 5 minutes ago
zmmzcm6xxzzo  flask-demo.2      master:5000/newapp:v1      worker-2  Running        Running 25 seconds ago
a292huwkzr23  flask-demo.3      master:5000/newapp:v1      worker-2  Running        Running 5 minutes ago
uuhbswsdvziq  flask-demo.4      master:5000/newapp:v1      worker-1  Running        Running 4 minutes ago
x4vrrs6fgpqz  flask-demo.5      master:5000/newapp:v1      worker-2  Running        Running 25 seconds ago
...

Deleting the Service:

$ docker service rm flask-demo

Deleting the Network:

$ docker network rm mynet

Leaving the Swarm:

To remove nodes from the swarm, run the following from the nodes:

[worker-1] $ docker swarm leave
[worker-2] $ docker swarm leave

Then we can list our nodes to get the nodes that we want to remove from the manager:

[master] $ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
j14mte3v1jhtbm3pb2qrpgwp6    worker-1  Down    Active
siqyf3yricsvjkzvej00a9b8h *  master    Ready   Drain         Leader
srl5yzme5hxnzxal2t1efmwje    worker-2  Down    Active

And removing the nodes from the managers list:

$ docker node rm worker-1
worker-1

$ docker node rm worker-2
worker-2

$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
siqyf3yricsvjkzvej00a9b8h *  master    Ready   Drain         Leader

Resources:

I hope this was useful, and I will continue writing some more content on Docker Swarm :D