The question: How can we share a common directory between our nodes easily? In other words, how can we make our app stateful in a cluster? I described the challenge over our Technical Challenge post.

User stories (requirement & specs)

As a DevOps hero:

  • As a DevOps hero, I'm looking for a NFS/ ZFS / GlusterFS server or whatever application that mounts a common directory between all my nodes.
  • As a DevOps hero, I want to have a common directory (not a docker volume) that all nodes can share. I suspect that having hundreds of docker volume will slow down over time. Something like /mnt/shared/permdata/
  • As a DevOps hero, I want to run the solution as a docker service create (...). No manual configs on each node and especially no hard IP to set up.
  • As a DevOps hero, I want to create a new node on my existing cluster. The data should sync automatically.

Per example, I would use it this way where permdata is the common directory :

  • /mnt/shared/permdata/app1/
  • /mnt/shared/permdata/app2/
  • /mnt/shared/permdata/bkp/
  • /mnt/shared/permdata/etc/

Solution

I use Resilio Sync since over a year and it's perfectly stable.

Performances: I tested it a lot! Most of the time, a file like a SQLite3 myapp.db or a picture get sync under 5 seconds. Sometimes it takes longer but most of the time it's fast.

Cons: The thing I don't like with this setup, is the fact that it use the public IP to sync. It's great as is very easy to use. But there is no need for this as I want to share stuff within my own cluster.

I want my service to use only the swarm network of my choice. Please let me know if there is a way in docker to force my service only to uses my private network (I use Digital Ocean).

Requirements

  • Have a docker swarm cluster running (3 nodes in my example).
  • Define our VAR:
MNT_SOURCE_RESILIO="/mnt/shared/permdata"
IMG_resilio="devmtl/resilio:2.6.3"
CTN_resilio1="node1"
CTN_resilio2="node2"
CTN_resilio3="node3"
  • Create this network:
NTW_RESILIO="ntw_resilio"

if [ ! "$(docker network ls --filter name=${NTW_RESILIO} -q)" ]; then
  docker network create --driver overlay --attachable --subnet 10.23.10.0/24 --opt encrypted ${NTW_RESILIO}
else
  echo "Network: ${NTW_RESILIO} already exist!"
fi
  • Create those labels:
docker node update --label-add nodeid=1 node1
docker node update --label-add nodeid=2 node2
docker node update --label-add nodeid=3 node3

How to

Step #1

Create a token:

docker run -dit \
-v $(pwd)/data:/data \
-p 33333:33333 \
"$IMG_resilio" && docker ps;

# find the secret
docker logs -f NAME; echo;

Find the TOKEN that looks like: A2QYBAQPK7SOP4O4ETFJEFHO5VLGHE747
What I do then is to copy this token on my nodes outside my git repo (along other secrets). In my case the file is here:

${MNT_DEPLOY_SETUP}/config/resilio/token

Delete this container. It was only to generate a token.

Step #2

Launch a Resilio Sync instance on each node.

Warning: Here we can't use --global because we need to configure different ports on each instance. This is because our service is using our public IP to sync data.

# First host
docker service rm ${CTN_resilio1}; \

docker service create \
  --name ${CTN_resilio1} --hostname ${CTN_resilio1} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 1' \
  --publish "33331:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}


# Second host
docker service rm ${CTN_resilio2}; \

docker service create \
  --name ${CTN_resilio2} --hostname ${CTN_resilio2} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 2' \
  --publish "33332:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

# Third host
docker service rm ${CTN_resilio3}; \

docker service create \
  --name ${CTN_resilio3} --hostname ${CTN_resilio3} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 3' \
  --publish "33333:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

Step #3

Now it's time to test it. From node1, put a file in ${MNT_SOURCE_RESILIO}. Wait a few seconds. Check on node2 and node3.

The file should appear quickly :)

Step #4

You might want to remove files and directories in the/mnt/shared/permdata/.sync/Archive directory as Resilio Sync will archive everything by default.

I have a crontab script that cleans this directory every hour.

Step #5

It's now time to build your own docker image with this Dockerfile. I shared my project on https://github.com/pascalandy/docker-resilio-sync

Please help

As mentioned earlier, please let me know if there is a way in docker to force my service only to uses my private network (I use Digital Ocean). There is no need to use the public network to do this operation.

Feel free to Buzz me on Twitter or in the Github repo.

Cheers!
Pascal