Container as an external hard drive: sharing a common directory (BitTorrent style) using Docker Swarm

FirePress Team -

The question: How can we share a common directory between our nodes easily? In other words, how can we make our app stateful in a cluster? I described the challenge over our Technical Challenge post.

Requirement & specs

As a DevOps hero:

  • As a DevOps hero, I'm looking for a private ZFS / GlusterFS server or whatever application that mounts a common directory between all my nodes.
  • As a DevOps hero, I want to have a common directory (not a docker volume) that all nodes can share. I suspect that having hundreds of docker volume will slow down over time. Something like /mnt/shared/permdata/
  • As a DevOps hero, I want to run the solution as a docker service create (...). No manual configs on each node and especially no hard IP to set up.
  • As a DevOps hero, I want to create a new node on my existing cluster. The data should sync automatically.
  • As a DevOps hero, I have a 3 nodes set up on docker swarm
  • As a DevOps hero, everything needs to happen via the CLI (no GUI operation)
  • No need external sync to the cloud (like AWS s3)
  • The traffic must use the swarm ingress, not via public traffic
  • Don't use a "volume" or a plug-in. The container must do the work of synching.

Per example, I would use it this way where permdata is the common directory :

/mnt/shared/permdata/app1/
/mnt/shared/permdata/app2/
/mnt/shared/permdata/bkp/
/mnt/shared/permdata/etc/

Solution

I use Resilio Sync since over a year and it's perfectly stable.

Performances: I tested it a lot! Most of the time, a file like a SQLite3 myapp.db or a picture get sync under 5 seconds. Sometimes it takes longer but most of the time it's fast.

Private network

UPDATE (2020-02-25): As expected, by default Resilio will use the public network. The good news is that you can set a configuration file to limit the network to eht0. You might know that on Digital Ocean the private network is happening over eth0. This way our network requirement is meet!

The stack

  • Have a docker swarm cluster running (3 nodes in my example).
  • Define our VAR:
MNT_SOURCE_RESILIO="/mnt/shared/permdata"
IMG_resilio="devmtl/resilio:2.6.3"
CTN_resilio1="node1"
CTN_resilio2="node2"
CTN_resilio3="node3"
  • Create this network:
NTW_RESILIO="ntw_resilio"

if [ ! "$(docker network ls --filter name=${NTW_RESILIO} -q)" ]; then
  docker network create --driver overlay --attachable --subnet 10.23.10.0/24 --opt encrypted ${NTW_RESILIO}
else
  echo "Network: ${NTW_RESILIO} already exist!"
fi
  • Create those labels:
docker node update --label-add nodeid=1 node1
docker node update --label-add nodeid=2 node2
docker node update --label-add nodeid=3 node3

How to

Step #1

Create a token:

docker run -dit \
-v $(pwd)/data:/data \
-p 33333:33333 \
"$IMG_resilio" && docker ps;

# find the secret
docker logs -f NAME; echo;

Find the TOKEN that looks like: A2QYBAQPK7SOP4O4ETFJEFHO5VLGHE747
What I do then is to copy this token on my nodes outside my git repo (along other secrets). In my case the file is here:

${MNT_DEPLOY_SETUP}/config/resilio/token

Delete this container. It was only to generate a token.

Step #2

Launch a Resilio Sync instance on each node.

Warning: Here we can't use --global because we need to configure different ports on each instance. This is because our service is using our public IP to sync data.

# First host
docker service rm ${CTN_resilio1}; \

docker service create \
  --name ${CTN_resilio1} --hostname ${CTN_resilio1} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 1' \
  --publish "33331:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}


# Second host
docker service rm ${CTN_resilio2}; \

docker service create \
  --name ${CTN_resilio2} --hostname ${CTN_resilio2} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 2' \
  --publish "33332:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

# Third host
docker service rm ${CTN_resilio3}; \

docker service create \
  --name ${CTN_resilio3} --hostname ${CTN_resilio3} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 3' \
  --publish "33333:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

Step #3

Now it's time to test it. From node1, put a file in ${MNT_SOURCE_RESILIO}. Wait a few seconds. Check on node2 and node3.

The file should appear quickly :)

Step #4

You might want to remove files and directories in the /mnt/shared/permdata/.sync/Archive directory as Resilio Sync will archive everything by default.

I have a crontab script that cleans this directory every hour.

Step #5

It's now time to build your own docker image with this Dockerfile. I shared my project over https://github.com/firepress-org/resilio-in-docker

Feel free to Buzz me on Twitter or in the Github repo.

Cheers!
Pascal