Persistent Storage in Docker Swarm

In this tutorial we will experiment with Docker Swarm Persistent Storage, backed by NFS using ContainX's Netshare Service

Get the dependencies:

If you have not provisioned a swarm using docker, have a look at setup a 3 node docker swarm post.

In order to mount NFS drives we need to install the following:

$ apt install nfs-common -y

Create NFS Server:

We have 2 options, we can either install a NFS Server (you can follow this post to setup a NFS Server ) or we can setup a NFS Server using Docker (explained below):

Prepare the NFS Home Directory:

$ mkdir /nfsdata

I would like to restrict NFS traffic to a specific interface, so I will use my private interface for my NFS traffic.

Create the NFS Server:

$ docker run --rm -itd --name nfs \
  --privileged \
  -v /nfsdata:/nfs.1 \
  -e SHARED_DIRECTORY=/nfs.1 \
  -p 10.0.2.15:2049:2049 \
  itsthenetwork/nfs-server-alpine:latest

Testing NFS

Test that NFS works, by creating a directory on the NFS home path and mount the NFS volume:

$ mkdir /nfsdata/test_folder
$ mount -v -t nfs4 -o vers=4,loud 10.0.2.15:/ /nfs_share
$ ls /nfs_share
test_folder

Now that we can see the directory via the NFS share, we can umount the NFS share, as the docker volume driver will mount the path inside the container:

$ umount /nfs_share

Install Netshare Docker Volume Driver

Install Netshare which will provide the NFS Docker Volume Driver:

$ wget https://github.com/ContainX/docker-volume-netshare/releases/download/v0.36/docker-volume-netshare_0.36_amd64.deb
$ dpkg -i docker-volume-netshare_0.36_amd64.deb
$ service docker-volume-netshare start

Create the NFS Volume

With NFS, everytime we create a NFS Volume, the path to that directory need to exit.

A couple of ways to create volumes:

Run a container with a persistent NFS backed volume (directory must exist)

Create the directory:

$ mkdir /nfsdata/foobar

Run the container:

$ docker run -i -t --volume-driver=nfs -v 10.0.2.15/foobar:/mount alpine /bin/sh

Inspect the filesystem in the container:

$ df -h | grep -e 'Filesystem\|mount'
Filesystem                Size      Used Available Use% Mounted on
10.0.2.15://foobar        9.1G      1.7G      6.8G  20% /mount

2. Create the volume and map the volume name:

Create the directory:

$ mkdir /nfsdata/foobar2

Create the docker volume:

$ docker volume create --driver nfs --name foobar2 -o share=10.0.2.15:/foobar2

Inspect the volume:

$ docker volume inspect foobar2
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "nfs",
        "Labels": {},
        "Mountpoint": "/var/lib/docker-volumes/netshare/nfs/foobar2",
        "Name": "foobar2",
        "Options": {
            "share": "10.0.2.15:/foobar2"
        },
        "Scope": "local"
    }
]

Run a container with the volume namespace:

$ docker run -i -t -v foobar2:/mount alpine /bin/sh

Demo: Using docker compose to persist nginx data in docker swarm

Create the directory:

$ mkdir /nfsdata/nginx_web

Create the index.html that the nginx container will use:

$ echo '<html>Hello, World!</html>' > /nfsdata/nginx_web/index.html

The docker-compose.yml

version: "3.7"
services:
  web:
    image: nginx
    volumes:
      - nginx.vol:/usr/share/nginx/html
    ports:
      - 80:80
    networks:
      - web

networks:
  web:
    driver: overlay
    name: web

volumes:
  nginx.vol:
    driver: nfs
    driver_opts:
      share: 10.0.2.15:/nginx_web

Deploy the application to the swarm:

$ docker stack deploy -c docker-compose.yml app

Test nginx:

$ curl -i http://localhost/
HTTP/1.1 200 OK
Server: nginx/1.17.1
Date: Tue, 23 Jul 2019 09:35:32 GMT
Content-Type: text/html
Content-Length: 27
Last-Modified: Tue, 23 Jul 2019 09:35:21 GMT
Connection: keep-alive
ETag: "5d36d4d9-1b"
Accept-Ranges: bytes

<html>Hello, World!</html>

We can see the content is being served from our persistent NFS volume, let's change the index.html:

$ echo '<html>hello, again :D</html>' > /nfsdata/nginx_web/index.html

And test again:

$ curl -i http://localhost/
HTTP/1.1 200 OK
Server: nginx/1.17.1
Date: Tue, 23 Jul 2019 09:36:16 GMT
Content-Type: text/html
Content-Length: 29
Last-Modified: Tue, 23 Jul 2019 09:36:14 GMT
Connection: keep-alive
ETag: "5d36d50e-1d"
Accept-Ranges: bytes

<html>hello, again :D</html>

Testing persistence, remove the stack and redeploy:

$ docker stack rm app
$ docker stack deploy -c docker-compose.yml app

Test again:

$ curl -i http://localhost/
HTTP/1.1 200 OK
Server: nginx/1.17.1
Date: Tue, 23 Jul 2019 09:38:33 GMT
Content-Type: text/html
Content-Length: 29
Last-Modified: Tue, 23 Jul 2019 09:36:14 GMT
Connection: keep-alive
ETag: "5d36d50e-1d"
Accept-Ranges: bytes

<html>hello, again :D</html>

Resources:

Thanks

If you enjoyed this post feel free to check out my website for more content at ruan.dev or follow me on twitter @ruanbekker