Our clients care about their Ghost updates. In this post, we will reveal how we update for all Ghost sites. Under the hood, many updates happen all the time. We maintain our own Ghost's docker image. Check it out on our GitHub release page.

At high level, we manage the backend for our clients. Our whole cluster runs on top of the public cloud, Linux OS, Docker Swarm, orchestrated services (containers), CI/CD, zero-downtime deployments and many other DevOps best-practices.

DevOps best practices are essential to us. Many checkpoints ensure our Ghost sites run smoothly.

Ghost updates

Now let's talk about the way we update Ghost for our clients. We usually wait 1-2 weeks before applying updates to our Ghost sites (and all our software packages really).

Here is an example showing why we don't automatically update Ghost as soon as it's released. This way, as a community, we can catch any emergency bugs that could emerge.

CI/CD

We take Continuous Integration and Continuous Deployment very seriously at FirePress. Only if you are curious to see what is happening with our Ghost builds, you can follow our Ghost builds here:
github.com/firepress-org/ghostfire/actions/workflows/build.yml

The master branch is probably the one you want to watch as it's the build your website will run on.

firepress-org/ghostfire

Docker image 🐳 for Ghost V3.x.x. This docker image is used on FirePress.org and play-with-ghost.com - firepress-org/ghostfire

GitHubfirepress-org

Best practices

This is how we carefully push every one of the Ghost updates. We use three kinds of tags:

1. site edge
2. site stable
3. site stable-hash (client's site)

It goes like this:

1. We use a repeatable Dockerfile declaration that executes everything that needs to happen to have a perfect installation of the software. No manual mistake can happen here as the Dockerfile contains the OS, libraries and the Ghost app.
2. A commit happens via Github. A webhook builds the docker image edge using Travis in a CI (continuous integration) system using the edge branch.
3. Within the CI, the system executes many tests. You can see one of them here. When the build is successful, the system tag this docker image as devmtl/ghostfire:edge.  If something fails, everything stops at this point, and we receive an email saying something has failed.
4. In our PROD environment, our CD (continuous deployment) checks every minute if a new edge image is available on the Dockerhub. This is almost magic! This applies only for the site edgetest.firepress.org (fake site, this is only to help you understand the concept)
5. We manually surf on edgetest.firepress.org . If we can navigate the site normally we consider the docker image edge as "passed. In the future, we might automate this part as well. Follow this issue for all details.
6. A commit happens via Github. A webhook builds the docker image stable using Travis in a CI (continuous integration) system using the stable branch.
7. Within the CI, the system executes many tests. You can see one of them here. When the build is successful, the system tag two specific tags: devmtl/ghostfire:stable and devmtl/ghostfire:stable-hash. If something fails, everything stops at this point, and we receive an email saying something has failed.
8. In our PROD environment, our CD (continuous deployment) checks every minute if a new stable image is available on Docker hub. It is still magical! This applies only for the site edgestable.firepress.org (fake site, this is only to help to reader understand the concept)
9. We manually surf on stabletest.firepress.org . If we can navigate the site normally we consider the docker image stable as "passed. In the future, we might automate this part as well. Follow this issue for all details.

Now all our tests are passed!

1. We are now ready to update all our client's sites using the Docker image devmtl/ghostfire:stable-hash.
2. We need to understand that the tags devmtl/ghostfire:stable and devmtl/ghostfire:stable-hash are actually the same docker image. This make our CD process so easy.
3. We update a configuration that specifies a tag (devmtl/ghostfire:stable-hash) that every site must run on. For a real-world example: see at the end of this log file and you will find: devmtl/ghostfire:2.23.3-30da41f
4. We commit the fact that every Ghost site must now run devmtl/ghostfire:2.23.3-30da41f
5. We launch the command which is a rolling update from the previous docker image to the newest one (devmtl/ghostfire:2.23.3-30da41f). This means zero downtime. Not even one second down.
6. If something is failing, it's probably not related to the docker container. It might be a network issue, a proxy issue, a load balancer issue, or something else.

Below, is a screenshot of three sites running in our cluster:

Our cluster

• There is no staging cluster. The staging is done in Github Action CI. We only have "staging sites" (edge, stable) along with all other sites we manage in the cluster.
• Our main cluster lives in New-York on top of one of the big Cloud providers.
• In the future, we might run two clusters in two regions. Per example, one in New-York and the other in Amsterdam. This will depend on our client's location. Fortunately, this is not a challenge at the moment as our sites run behind a powerful CDN powered by Cloudflare.

Conclusion

You can see how we do everything we can to avoid human errors and to ensure that you are always running a fresh build.

It's good to understand that we never update the existing docker image. We always build a new one from scratch. This way, all patches and security fixes are applied onto your Ghost site during every update.

Cheers!
Pascal — FirePress Team 🔥📰

The trick is... there is no crontab on Mac since 6-7 years. You must use launchctl to schedule a "cron job".

In my case, I simply want to run a bash script for my local backup. I guess that if you are confortable running cron jobs on Linux, you will be at home with these instructions.

How To

launchctl unload ~/Library/LaunchAgents/com.pascalandy.macbackup.plist

cd ~/Library/LaunchAgents
nano com.pascalandy.macbackup.plist

<INSERT XML CODE BELOW>
<SAVE AND QUIT NANO>

launchctl load ~/Library/LaunchAgents/com.pascalandy.macbackup.plist
launchctl list | grep 'pascalandy'

Code example

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>

        <key>Label</key>
        <string>com.pascalandy.macbackup</string>

        <key>ProgramArguments</key>
        <array>
            <string>/bin/bash</string>
            <string>/Volumesxyz/pascalandy/macbkp/runup.sh</string>
        </array>

        <key>LowPriorityIO</key>
        <true/>

        <key>Nice</key>
        <integer>1</integer>

        <key>StartCalendarInterval</key>
        <dict>
            <key>Hour</key>
            <integer>11</integer>
            <key>Minute</key>
            <integer>39</integer>
        </dict>

    </dict>
</plist>

(XML format)

Sources

If you need more blabla, there you go :)

• https://alvinalexander.com/mac-os-x/mac-osx-startup-crontab-launchd-jobs/
• https://thejandroman.wordpress.com/2013/02/13/introduction-to-launchd/
• https://www.launchd.info/
• https://firepress.org/en/how-to-run-bash-scripts-like-a-crontab-for-mac-big-sur/

To perform an upgrade on our proxies, we need to regenerate a few SSL certificates. This action might impact your website and, it would cause downtime of a few minutes on your website.

Please avoid to update your website during the maintenance window. There is no action required on your part.

Benefit:
The upgrade will also allow our client to point their domain name to a CNAME. It's the Option C as detailed here: https://firepress.org/en/how-can-i-configure-my-domain-or-dns-to-firepress-servers/

Maintenance window:
this Saturday night, Montreal time zone (UTC -5 EST)

Start: 2020-11-28 20h00
End: 2020-11-29 04h00

Thank you for your patience. If you face any issues, please reach out to us.

We had some hickup on that day. We were running after our tail try to understand what could have gone wrong on our side.

Finally, it was on Cloudflare's side and it was a human mistake. Here is what happened: https://blog.cloudflare.com/cloudflare-outage/

Not all FirePress's clients were affected as not all sites under Cloudflare's were concerned.

In any case, we are sorry for the trouble that may have caused you.

We also updated our process to better react to such event in the future.

Cheers!
Pascal Andy

The question: How can we share a common directory between our nodes easily? In other words, how can we make our app stateful in a cluster? I described the challenge over our Technical Challenge post.

Requirement & specs

As a DevOps hero:

• As a DevOps hero, I'm looking for a private ZFS / GlusterFS server or whatever application that mounts a common directory between all my nodes.
• As a DevOps hero, I want to have a common directory (not a docker volume) that all nodes can share. I suspect that having hundreds of docker volume will slow down over time. Something like /mnt/shared/permdata/
• As a DevOps hero, I want to run the solution as a docker service create (...). No manual configs on each node and especially no hard IP to set up.
• As a DevOps hero, I want to create a new node on my existing cluster. The data should sync automatically.
• As a DevOps hero, I have a 3 nodes set up on docker swarm
• As a DevOps hero, everything needs to happen via the CLI (no GUI operation)
• No need external sync to the cloud (like AWS s3)
• The traffic must use the swarm ingress, not via public traffic
• Don't use a "volume" or a plug-in. The container must do the work of synching.

Per example, I would use it this way where permdata is the common directory :

/mnt/shared/permdata/app1/
/mnt/shared/permdata/app2/
/mnt/shared/permdata/bkp/
/mnt/shared/permdata/etc/

Solution

I use Resilio Sync since over a year and it's perfectly stable.

Performances: I tested it a lot! Most of the time, a file like a SQLite3 myapp.db or a picture get sync under 5 seconds. Sometimes it takes longer but most of the time it's fast.

Private network

UPDATE (2020-02-25): As expected, by default Resilio will use the public network. The good news is that you can set a configuration file to limit the network to eht0. You might know that on Digital Ocean the private network is happening over eth0. This way our network requirement is meet!

The stack

• Have a docker swarm cluster running (3 nodes in my example).
• Define our VAR:

MNT_SOURCE_RESILIO="/mnt/shared/permdata"
IMG_resilio="devmtl/resilio:2.6.3"
CTN_resilio1="node1"
CTN_resilio2="node2"
CTN_resilio3="node3"

• Create this network:

NTW_RESILIO="ntw_resilio"

if [ ! "$(docker network ls --filter name=${NTW_RESILIO} -q)" ]; then
  docker network create --driver overlay --attachable --subnet 10.23.10.0/24 --opt encrypted ${NTW_RESILIO}
else
  echo "Network: ${NTW_RESILIO} already exist!"
fi

• Create those labels:

docker node update --label-add nodeid=1 node1
docker node update --label-add nodeid=2 node2
docker node update --label-add nodeid=3 node3

How to

Step #1

Create a token:

docker run -dit \
-v $(pwd)/data:/data \
-p 33333:33333 \
"$IMG_resilio" && docker ps;

# find the secret
docker logs -f NAME; echo;

Find the TOKEN that looks like: A2QYBAQPK7SOP4O4ETFJEFHO5VLGHE747
What I do then is to copy this token on my nodes outside my git repo (along other secrets). In my case the file is here:

${MNT_DEPLOY_SETUP}/config/resilio/token

Delete this container. It was only to generate a token.

Step #2

Launch a Resilio Sync instance on each node.

Warning: Here we can't use --global because we need to configure different ports on each instance. This is because our service is using our public IP to sync data.

# First host
docker service rm ${CTN_resilio1}; \

docker service create \
  --name ${CTN_resilio1} --hostname ${CTN_resilio1} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 1' \
  --publish "33331:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}


# Second host
docker service rm ${CTN_resilio2}; \

docker service create \
  --name ${CTN_resilio2} --hostname ${CTN_resilio2} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 2' \
  --publish "33332:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

# Third host
docker service rm ${CTN_resilio3}; \

docker service create \
  --name ${CTN_resilio3} --hostname ${CTN_resilio3} \
  --network ${NTW_RESILIO} --replicas "1" \
  --restart-condition "any" --restart-max-attempts "20" \
  --reserve-memory "192M" --limit-memory "512M" \
  --limit-cpu "0.333" \
  --constraint 'node.labels.nodeid == 3' \
  --publish "33333:33333" \
  -e RSLSYNC_SECRET=$(cat ${MNT_DEPLOY_SETUP}/config/resilio/token) \
  --mount type=bind,source=${MNT_SOURCE_RESILIO},target=/data \
  ${IMG_resilio}

Step #3

Now it's time to test it. From node1, put a file in ${MNT_SOURCE_RESILIO}. Wait a few seconds. Check on node2 and node3.

The file should appear quickly :)

Step #4

You might want to remove files and directories in the /mnt/shared/permdata/.sync/Archive directory as Resilio Sync will archive everything by default.

I have a crontab script that cleans this directory every hour.

Step #5

It's now time to build your own docker image with this Dockerfile. I shared my project over https://github.com/firepress-org/resilio-in-docker

Feel free to Buzz me on Twitter or in the Github repo.

Cheers!
Pascal

Hi!

First you might be interested by our Roadmap. This page is about deep technical & architectural challenge we have.

Sharing challenges feels like the right thing to do as I get so much from the open-source community. If this can help people to better understand what we are building here, I'm glad to share it.

NOTE: The text below is written by a voice recognition software. It's might look funny and is not edited by a human.

Table of Content

• Backlog
• Dropped
• Up and Running
• Get involved

Backlog

🙊 Container as an external hard drive

User stories / specs

As a DevOps hero:

• As a DevOps hero, I'm looking for a nfs/zfs/GlusterFS or whatever application that mounts a common directory between all my nodes.
• This needs to run as a docker service create XYZ --global with Docker Swarm. No manual configs on each node and no hard IP to set up.
• As a DevOps hero, I want to create a new node on my existing cluster. The data should sync automatically.
• As a DevOps hero, I want to have a common directory (not a docker volume) that all nodes can share. Something like /mnt/shared/permdata/

Per example, I would use it this way:

• /mnt/shared/permdata/app1/
• /mnt/shared/permdata/app2/
• /mnt/shared/permdata/bkp/
• /mnt/shared/permdata/etc/

Work around

At the moment I use Resilio which is great. The thing I don't like is the fact that it use the the public network to sync. There is no need for this. I want my service to use only the swarm network of my choice.

Maybe I could force resilio to sync only within an overlay network?

by: Pascal Andy / 2019-02-26

🙊 Cluster crash mitigation

EDIT: 2019-02-26_15h05: The scenario below is well managed. It's not in prod yet only because we don't have a lot of nodes at the moment. It would be too costly for now. But everything is in place to make it work very quickly.

Scenario:

This is a big one. Let's say a whole cluster is not available for 6 hours. Whatever the reason. Shall we, as a business, cry on Twitter that our server vendor are down? Absolutely not! Remember the S3 crash in April 2017? Shit happens and I don't want this to happen to us at FirePress.

The idea here is that we would have two independent clusters running in two zones (data centre).

• 50% of our clients are in NYC
• 50% of our clients are in AMS

Let's say NYC crash. Fuck. OK no panic.

Deploy 100% of our clients to AMS.

The challenge is to this very quickly. Database merging + picture merging.

Then, went things are back to normal, redistribute 50%/50%.

With this setup, it also allows an easy transition from one cluster to a new one. I love it. Don't patch. Scrap and start from scratch.

🙊 Roadmap

https://trello.com/b/0fCwwzqc/firepress-roadmap

Go back to Table of content

Dropped

Caching website / blogs

• Challenge — Add a Varnish caching container for each blog (or maybe one for every domain we host??)
• CMO, a request goes to Traefik CNT > Ghost CTN > MySQL CTN
• FMO, I want Traefik CNT > Varnish Cache > (if contain is not cached...) > Ghost CTN > MySQL CTN

Minio storage for our private Docker registry

• All nodes in the cluster shall have access to Minio bucket
• Would be nice to use Backblaze B2 as storage provider - wip
• To consider | https://github.com/cloudflavor/miniovol
• Storage pricing is key. No AWS S3.
• Backblaze is the best deal at the moment. I use them to do our back up.
• maybe REX-Ray
• To test | https://twitter.com/askpascalandy/status/862271673072058368
• https://github.com/codedellemc/labs
• maybe Portworx and Minio together
• https://www.youtube.com/watch?v=5gRQN9WxsIk

Deploy a HA MySQL database

• 2019-02-26: See https://gist.github.com/pascalandy/5735bcae8257e861f29e06da46754aef
• I use Percona and I should be able to do HA. I don't know how yet
• Galera Cluster looks promising
• Mysql 8 will support HA natively
• At the moment, I run one instance of Percona (no HA). Resilio syncing a common directory between 3 nodes.
• Still trying to find a solution to easily run a MySQL cluster master-master-master
• To consider | https://github.com/pingcap/tidb
• This setup looks promising but it’s not quite perfect yet.
• http://severalnines.com/en/mysql-docker-deploy-homogeneous-galera-cluster-etcd
• https://github.com/pingcap/docs/blob/master/op-guide/docker-deployment.md

Monitoring our DB | PMM

• https://www.percona.com/en/2017/04/21/percona-monitoring-management-1-1-3-now-available/

ChatOps

DROPPED. Ops will use a terminal. That's it.

It would be nice to use Slack as a terminal. Why is that?? Here is my use case.

I want to let none-technical folks (the operations) run Docker stack without having to setup their user/pass/local environment and all the pain that come with welcoming a new user in your DevOps stuff. I assume I could prevent from doing some actions as well like rm *.

Go back to Table of content

Up and running

To see how we roll (technically speaking) at FirePress, please check the post What kind of Back-End drives FirePress.

In short we have hosting challenges. Think static website and blog/CMS (Ghost) sites. This site is actually running within a container at http://firepress.org/en/. The home page is running into another one at http://firepress.org/.

• ✅ Our stack is cloud agnostic. No AWS/Azure/Google locked in.
• ✅ We use Ubuntu servers a deploy them via CLI Docker Machine
• ✅ We configure our servers via a bash script / docker-machine. No need for teraform at the moment but probably will some day.
• ✅ We set UFW rules to work along Docker
• ✅ We run services docker service create (well 95% of the time).
• ✅ We use Resilio service to share a common folder between all nodes. Looking to switch… see below.
• ✅ Reverse proxy to redirection public traffic
• ✅ Docker label and deploy services against those constraints

✅ Fancy bash script to launch services like:

• Traefik
• Percona (MySQL)
• Ghost
• Nginx
• Portainer
• Sematext
• rClone

Most containers are built on Alpine.

• ✅ We deploy each website via an unique ID
• ✅ Generate dynamic landing page via a script from an HTML template. Nothing fancy yet, but great at this stage.

✅ Our back up processes are solid.

• Via cron
• Internval: every 4 hours, every day
• Compressed and encrypt before going outside the cluster on Backblaze B2.
• Notified in Slack when the backup is done
• Keeping only the last 2 backup on the DB node
• Swarm (raft) is also backed up
• ✅ Cron docker system prune --all --force on each node
• ✅ Cron back up the Swarm Raft

✅ Docker build

• Highly standardized for all containers
• Tagging edge, stable, version are made automatically. We build our containers simply by running ./builder.sh + directory name
• Versioning is A1. We use tags: edge and stable
• ✅ We deploy our web app with a PathPrefix (Traefik)
• mycie.com/green/
• mycie.com/blue/
• mycie.com/yellow/
• We use Cloudflare CLI - Create, update, delete | Zone, A, CNAME etc via flarectn which run within a sporadic container

✅ We contribute to making Docker a better place

• Feature Request: Show --global instance numbers when docker service
• Fixed — https://github.com/moby/moby/issues/27670
• Scheduler limits the # of ctn at 40 per nodes worker (overlay network limit is 252 ctn) | Swarm 1.12.1
• Fixed — https://github.com/moby/moby/issues/26702

Monitoring stack Swarmprom / portainer

• Metrics | Collects, processes, and publishes metrics
• Intel Snap | Collects, processes, and publishes metrics
• InfluxDB | Stores metrics
• Grafana | Displays metrics visually
• Logs ELK (ElasticSearch, Logstash, Kibana)
• Alerts management (i.e. one node is not responsive)
• Monitoring Percona Mysql performance DB (in docker of course)

Traefik config

• Traefik is a beast. So many configs!
• Traefik allows me to automatically create https for each site. But I can’t make it work along Cloudflare service. It’s one or the other. I’m screwed so I don’t use SSL at the moment.
• Test ACME renewal

DNS load balance BEFORE hitting the swarm cluster

• Challenge — At the moment, Cloudflare point to to ONE node. If this node crash, all our site goes down !
• Cloudflare are working on their load balancing solution but let's be proactive. See this ticket.
• We need a health check to see if our 3 managers are health and do a round robin sticky session between them. If one manager is not healthy, the round-robin system shall stop sending traffic to this node. If node Leader 1 is down, the system shall point traffic to node Leader 2 or 3 (health check).

Zero-downtime deployments with rolling upgrades

• Will be fixed by the docker team
• https://github.com/moby/moby/issues/30321

Find the best practice to update each node

• At the moment the docker deamon needs to restart... and the DB goes down for 1-2 minutes

Redirect path to domain.com/web'/'

• Known issue with Traefik See https://github.com/containous/traefik/issues/1123#issue-205597693
• Should be fix on traefik with 1.3. See PR

Deploying servers at scale

• Build a Packer / Terraform routine to deploy new nodes (see also SCW Builder)
• Minimize manual processes (of running bash scripts) to setup up Docker Swarm join / Gluster, UFW rules for private networks
• Better use of Docker-machine so I can use eval more efficient instead of switching between terminal windows.

CICD

• Of course one day it will make sense to get there
• I don't feel the need for this at the moment, the docker workflow by itself is solid enough
• Would be great to rebuild image every night

Go back to Table of content

Get involved

If you have solid skills 🤓 with Docker Swarm, Linux bash and the gang* and you would love to help a startup to launch 🔥 a solid project, I would love to get to know you 🍻. Buzz me 👋 on Twitter @askpascalandy. You can see the things that are done and the things we have to do here.

I’m looking for bright and caring people to join this journey with me.

To see how we roll (technically speaking) at FirePress, please check the post What kind of Back-End drives FirePress.

We are hosting between 30 to 60 websites/en/services at any given moment. Not so much at this point as we are in the Beta phase. I’m looking to define an official SLA for our stack.

In short we have hoster challenges. Think static website and blog/CMS (Ghost) sites. This site is actually running within a container at firepress.org/en/.

Thanks in advance!
Pascal

Go back to Table of content

Here is a checklist for people maintaining their own Ghost themes based on themes we maintain at FirePress.

When a new release is available, just override the themes but not those parts. It avoids the pain of comparing each of the files and minimizes human error.

1) Custom files

/partials/custom_footer.hbs
/partials/custom_header.hbs
/package.json

2) Custom directory

/assets/css_firepress/

3) Custom default.hbs

In default.hbs, we do reference:

{{>custom_header}}
{{>custom_footer}}
custom <footer class="site-foot">

Then, follow this checklist available on our Roadmap.

Here is a great video that shows how to use TMUX on mac and how to use shortcuts, commands, scripts, configurations and more.

Goal

Use Docker secrets to avoid saving sensitive credentials within your image or passing them directly on the command line.

Source: https://docs.docker.com/engine/swarm/secrets/#about-secrets

#hashtag:

/ origin: RUN FROM ON NODE MANAGER

/ on_node: RUN FROM ON NODE WHERE THE SERVICE WAS DEPLOYED

Basic Redis example

1) Create a secret #origin

echo "mysecret1b1ee244779d3b7a90bc80be3721fac5f26b350480680" | docker secret create SECRET_POSTGRES_ROOT -;

2) RM a secret #origin

docker secret rm SECRET_POSTGRES_ROOT -;

3) Create a demo container #origin

docker service  create --name="redis" --secret="SECRET_POSTGRES_ROOT" redis:alpine;

4a) Find the container ID #on_node

docker ps --filter name=redis -q;

4b) Ensure that 'SECRET_POSTGRES_ROOT' is available #on_node

docker exec $(docker ps --filter name=redis -q) ls -l /run/secrets;

4c) Show value for 'SECRET_POSTGRES_ROOT' #on_node

docker exec $(docker ps --filter name=redis -q) cat /run/secrets/SECRET_POSTGRES_ROOT;

5) Remove a secret #origin

docker service update --secret-rm="SECRET_POSTGRES_ROOT" redis

Ensure it's removed

See Step 4c

6) Add secret to an existing container #origin

docker service update --secret-add="SECRET_POSTGRES_ROOT" redis;

Ensure it's available

See Step 4c

7) Clean up stuff from this demo

docker service rm redis mysql;
docker secret rm SECRET_POSTGRES_ROOT;

Wordpress and Mysql example

Create a random password #origin

openssl rand -base64 20 | docker secret create mysql_password -;
openssl rand -base64 20 | docker secret create mysql_root_password -;
docker secret ls;

Create network #origin

docker network create -d overlay mysql_private;

Create mysql service

docker service create \
 --name mysql \
 --replicas 1 \
 --network mysql_private \
 --mount type=volume,source=mysql-data,destination=/var/lib/mysql \
 --secret source=mysql_root_password,target=mysql_root_password \
 --secret source=mysql_password,target=mysql_password \
 -e MYSQL_ROOT_PASSWORD_FILE="/run/secrets/mysql_root_password" \
 -e MYSQL_PASSWORD_FILE="/run/secrets/mysql_password" \
 -e MYSQL_USER="wordpress" \
 -e MYSQL_DATABASE="wordpress" \
 mysql:latest;

docker service ls;
docker service ps mysql;

Create wordpress service

docker service create \
     --name wordpress \
     --replicas 1 \
     --network mysql_private \
     --publish 80:80 \
     --mount type=volume,source=wpdata,destination=/var/www/html \
     --secret source=mysql_password,target=wp_db_password,mode=0400 \
     -e WORDPRESS_DB_USER="wordpress" \
     -e WORDPRESS_DB_PASSWORD_FILE="/run/secrets/wp_db_password" \
     -e WORDPRESS_DB_HOST="mysql:3306" \
     -e WORDPRESS_DB_NAME="wordpress" \
     wordpress:latest;

/// mode - specifies that the secret is not group-or-world-readable, by setting the mode to 0400.

docker service ls;
docker service ps wordpress;

Log on the wordpress site

127.0.0.1:80

Rotate a secret¶

Create a 2ND random password #origin

openssl rand -base64 44 | docker secret create mysql_password_v2 -;
docker secret ls;

See what is the password we just created

docker service create --name="checkthis" --secret="mysql_password_v2" alpine sleep 20;
echo; sleep 3;
docker exec $(docker ps --filter name=checkthis -q) cat /run/secrets/mysql_password_v2;
echo; sleep 3;
docker service rm checkthis; echo;

Update the MySQL service

docker service update \
--secret-rm mysql_password mysql;

docker service update \
--secret-add source=mysql_password,target=old_mysql_password \
--secret-add source=mysql_password_v2,target=mysql_password \
mysql;

/// mysql is restarting

docker service ls;
docker service ps mysql;

/// Even though the MySQL service has access to both the old and new secrets now, the MySQL password for the WordPress user has not yet been changed.

Update MySQL password for the wordpress user using the mysqladmin

docker exec $(docker ps --filter name=redis -q)
docker exec $(docker ps --filter name=mysql -q) \
bash -c 'mysqladmin --user=wordpress --password="$(< /run/secrets/old_mysql_password)" password "$(< /run/secrets/mysql_password)"';

Update the wordpress service to use the new password

docker service update \
--secret-rm mysql_password \
--secret-add source=mysql_password_v2,target=wp_db_password,mode=0400 \
wordpress;

/// wordpress is restarting

docker service ls;
docker service ps wordpress;

/// Verify that WordPress works by browsing to http://localhost:30000/ on any swarm node again

Revoke access to the old secret from the MySQL service and remove the old secret from Docker.

docker service update \
--secret-rm mysql_password \
mysql;

docker secret rm mysql_password;

Clean up stuff from this demo

docker service rm wordpress mysql;

docker volume rm mydata wpdata;

docker secret rm mysql_password_v2 mysql_root_password;

/// That was easy!

See ya soon!
Pascal Andy | Twitter

[

P.S. If you have solid skills 🤓 with Docker Swarm, Linux and the things mention here and you would love 💚 to help a startup to launch 🔥 a solid project, I would love to get to know you 🍻. Buzz me 👋 on Twitter @askpascalandy. You can see the things that are done and the things we have to do here.

The goal

I want to share/sync a common folder between 4 nodes.
You know like dropbox but without a 3td party server of course.
Let's see if (Minio Erasure Code) can help.

This doc is not on Minio website yet but it really helped me.

Create the folder to share between our 4 nodes:

Run this on all nodes:

rm -rf /mnt/minio;
mkdir -p /mnt/minio/dev-e;
cd /mnt/minio/dev-e; ls -AlhF;

About my path SOURCE:

• mnt is for things shared
• minio is the driver or the applications used to share
• dev-d is my cluster ID. It could be prod-a, prod-b, dev-b ...

Network

Run this the leader node:

docker network create --driver overlay ntw_minio

Deploying 4 instances (Minio Erasure Code)

Run this the leader node:

Create your own MINIO_ACCESS_KEY and MINIO_SECRET_KEY values!

• Ensure access key = 5 to 20 characters
• Ensure secret key = 8 to 40 characters

### Start service 1
CTN_NAME=minio_N01
SVR_NAME=node1
ENV_PORT=9001
\
docker service create \
--name "$CTN_NAME" \
--network "ntw_minio" \
--replicas "1" \
-p "$ENV_PORT":9000 \
--constraint node.hostname=="$SVR_NAME" \
--mount type=bind,src=/mnt/minio/dev-e,dst=/export \
-e "MINIO_ACCESS_KEY=A5a0a87b725552daXd" \
-e "MINIO_SECRET_KEY=369f5e7b4a41e25452c353D629a24c372b62c90" \
minio/minio \
server \
http://minio_N01/export \
http://minio_N11/export \
http://minio_N12/export \
http://minio_N13/export


### Start service 2
CTN_NAME=minio_N11
SVR_NAME=node2
ENV_PORT=9002
\
docker service create \
--name "$CTN_NAME" \
--network "ntw_minio" \
--replicas "1" \
-p "$ENV_PORT":9000 \
--constraint node.hostname=="$SVR_NAME" \
--mount type=bind,src=/mnt/minio/dev-e,dst=/export \
-e "MINIO_ACCESS_KEY=A5a0a87b725552daXd" \
-e "MINIO_SECRET_KEY=369f5e7b4a41e25452c353D629a24c372b62c90" \
minio/minio \
server \
http://minio_N01/export \
http://minio_N11/export \
http://minio_N12/export \
http://minio_N13/export

### Start service 3
CTN_NAME=minio_N12
SVR_NAME=node3
ENV_PORT=9003
\
docker service create \
--name "$CTN_NAME" \
--network "ntw_minio" \
--replicas "1" \
-p "$ENV_PORT":9000 \
--constraint node.hostname=="$SVR_NAME" \
--mount type=bind,src=/mnt/minio/dev-e,dst=/export \
-e "MINIO_ACCESS_KEY=A5a0a87b725552daXd" \
-e "MINIO_SECRET_KEY=369f5e7b4a41e25452c353D629a24c372b62c90" \
minio/minio \
server \
http://minio_N01/export \
http://minio_N11/export \
http://minio_N12/export \
http://minio_N13/export


### Start service 4
CTN_NAME=minio_N13
SVR_NAME=node4
ENV_PORT=9004
\
docker service create \
--name "$CTN_NAME" \
--network "ntw_minio" \
--replicas "1" \
-p "$ENV_PORT":9000 \
--constraint node.hostname=="$SVR_NAME" \
--mount type=bind,src=/mnt/minio/dev-e,dst=/export \
-e "MINIO_ACCESS_KEY=A5a0a87b725552daXd" \
-e "MINIO_SECRET_KEY=369f5e7b4a41e25452c353D629a24c372b62c90" \
minio/minio \
server \
http://minio_N01/export \
http://minio_N11/export \
http://minio_N12/export \
http://minio_N13/export

docker service ls

docker service ps minio_N01

ID            NAME      IMAGE                                     NODE   DESIRED STATE  CURRENT STATE          ERROR  PORTS
bx6ayw4hw43q  minio1.1  minio/minio:RELEASE.2017-01-25T03-14-52Z  node1  Running        Running 2 minutes ago
[node1] (local) root@10.0.17.3 /mnt/minio/dev-d

docker service ps minio_N11
ID            NAME      IMAGE                                     NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
ommy53chajmh  minio2.1  minio/minio:RELEASE.2017-01-25T03-14-52Z  node2  Running        Running 51 seconds ago
[node1] (local) root@10.0.17.3 /mnt/minio/dev-d

docker service ps minio_N12
ID            NAME      IMAGE                                     NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
iykg3oeo56mh  minio3.1  minio/minio:RELEASE.2017-01-25T03-14-52Z  node3  Running        Running 33 seconds ago
[node1] (local) root@10.0.17.3 /mnt/minio/dev-d

docker service ps minio_N13
ID            NAME      IMAGE                                     NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
wmf3aim5f3gr  minio4.1  minio/minio:RELEASE.2017-01-25T03-14-52Z  node4  Running        Running 21 seconds ago
[node1] (local) root@10.0.17.3 /mnt/minio/dev-d

logs from minio1

ctn_NAME=minio_N01 && \
ctnID=$(docker ps -q --filter label=com.docker.swarm.service.name=$ctn_NAME) && \
docker logs --follow $ctnID

ctn_NAME=minio_N11 && \
ctnID=$(docker ps -q --filter label=com.docker.swarm.service.name=$ctn_NAME) && \
docker logs --follow $ctnID

ctn_NAME=minio_N12 && \
ctnID=$(docker ps -q --filter label=com.docker.swarm.service.name=$ctn_NAME) && \
docker logs --follow $ctnID

ctn_NAME=minio_N13 && \
ctnID=$(docker ps -q --filter label=com.docker.swarm.service.name=$ctn_NAME) && \
docker logs --follow $ctnID


lStorage()]"
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 9s)
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 24s)
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 40s)
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 56s)
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 1m9s)
Initializing data volume. Waiting for minimum 3 servers to come online. (elapsed 1m29s)
Disk minio4:9000:/minio/storage/export is still unreachable, with error disk not found
Initializing data volume for first time. Waiting for other servers to come online (elapsed 1m51s)

Initializing data volume for the first time.
[01/04] http://minio1:9000/export - 10 GiB online
[02/04] http://minio2:9000/export - 10 GiB online
[03/04] http://minio3:9000/export - 10 GiB online
[04/04] http://minio4:9000/export - 10 GiB online

Endpoint:  http://10.255.0.8:9000  http://10.255.0.7:9000  http://172.19.0.3:9000  http://10.0.1.3:9000  http://10.0.1.2:9000  http://127.0.0.1:9000
AccessKey: A18d29a3a0256b1e606
SecretKey: Sdf128a527d40fd6811df3f0a72136b9e9201
Region:    us-east-1
SQS ARNs:  <none>

Browser Access:
   http://10.255.0.8:9000  http://10.255.0.7:9000  http://172.19.0.3:9000  http://10.0.1.3:9000  http://10.0.1.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://10.255.0.8:9000 A18d29a3a0256b1e606 Sdf128a527d40fd6811df3f0a72136b9e9201

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide

Drive Capacity: 18 GiB Free, 20 GiB Total
Status:         4 Online, 0 Offline. We can withstand [2] more drive failure(s).

Status 1)

The services are running good.

Create a bucket

• Open a new tab on your browser
• Go to: http://ip10_0_25_6-9001.play-with-docker.com/minio
• Enter credits
• Create bucket 'tester'
• Upload a picture 'animated-good-job.gif' from the browser

On your 4 nodes, check if the file is there:

$ cd /mnt/minio/dev-d/tester; ls -AlhF;
total 0
drwxr-xr-x    2 root     root          35 Feb 14 00:31 animated-good-job.gif/
[node1] (local) root@10.0.25.3 /mnt/minio/dev-d/tester

Status 2)

When uploading a file from the web GUI, all nodes sync the files as expected. Good!

2/2 Testing file sharing by creating a file from the nodes

#### from node1, Create dummy files (unit test)
FROM_NODE=node1; \
FILE_SIZE=11M; \
\
LENGTH="8"; \
RAND_STRING=null; \
STEP1=$((RANDOM%975+211)); \
STEP2=$(openssl rand -base64 "$STEP1"); \
STEP3=$(echo "$STEP2" | shasum -a 512 | head -c "$LENGTH"); echo; \
RAND_STRING="$STEP3"; STEP1="null"; STEP2="null"; STEP3="null"; \
echo "$RAND_STRING"; echo; \
\
cd /mnt/minio/dev-d/tester; \
echo "Create a dummy text file:"; echo; \
pwd; ls -AlhF; du -sh; echo; \
WHEN="$(date +%Y-%m-%d_%H-%M-%S)"; \
echo "Created from $FROM_NODE - $WHEN" >> "$FROM_NODE"_"$RAND_STRING".txt; \
pwd; ls -AlhF; du -sh; echo; cat "$FROM_NODE".txt; echo; echo; \
\
pwd; ls -AlhF; du -sh; echo; \
WHEN="$(date +%Y-%m-%d_%H-%M-%S)"; \
dd if=/dev/zero of="$FROM_NODE"_"$RAND_STRING".dat  bs=$FILE_SIZE  count=1; \
pwd; ls -AlhF; du -sh; echo; \
watch -d -n 1 ls -AlhF;

Then ...

#### from node2, Create dummy files (unit test)
FROM_NODE=node2; \
FILE_SIZE=12M; \
\
LENGTH="8"; \h
RAND_STRING=null; \
STEP1=$((RANDOM%975+211)); \
STEP2=$(openssl rand -base64 "$STEP1"); \
STEP3=$(echo "$STEP2" | shasum -a 512 | head -c "$LENGTH"); echo; \
RAND_STRING="$STEP3"; STEP1="null"; STEP2="null"; STEP3="null"; \
echo "$RAND_STRING"; echo; \
\
cd /mnt/minio/dev-d/tester; \
echo "Create a dummy text file:"; echo; \
pwd; ls -AlhF; du -sh; echo; \
WHEN="$(date +%Y-%m-%d_%H-%M-%S)"; \
echo "Created from $FROM_NODE - $WHEN" >> "$FROM_NODE"_"$RAND_STRING".txt; \
pwd; ls -AlhF; du -sh; echo; cat "$FROM_NODE".txt; echo; echo; \
\
pwd; ls -AlhF; du -sh; echo; \
WHEN="$(date +%Y-%m-%d_%H-%M-%S)"; \
dd if=/dev/zero of="$FROM_NODE"_"$RAND_STRING".dat  bs=$FILE_SIZE  count=1; \
pwd; ls -AlhF; du -sh; echo; \
watch -d -n 1 ls -AlhF;

from node3, Create dummy files (unit test)

You get the pattern at this point :)

from node4, Create dummy files (unit test)

You get the pattern at this point :)

Status 3)

Files are NOT SYNCED when they are created from the nodes. Is it normal?

Asking for help on Slack

Original conversation is here.

Hello folks!

Regarding Minio Erasure Code Mode,
I want to share/sync a common folder between 4 nodes using Erasure Code Mode.
You know like dropbox (but without a 3td party main server of course).

I took many hours to test this setup and this is my conclusion:

• When uploading a file from the web GUI, all nodes sync the files as expected. Good!
• But files are NOT SYNCED when they are created from the nodes. Damm :-/

May I ask your help here?
https://github.com/minio/minio/issues/3713#issuecomment-279573366

Cheers!

Answers on Slack!

y4m4b4 [8:18 PM]
mounting a common DIR you can either use MinFS or S3FS

[8:18]
which would mount the relevant bucket on the nodes..

pascalandy [8:18 PM]
OK tell me about it :)))

y4m4b4 [8:18 PM]
https://github.com/minio/minfs#docker-simple
minio/minfs: A network filesystem client to connect to Minio and Amazon S3 compatible cloud storage servers
minfs - A network filesystem client to connect to Minio and Amazon S3 compatible cloud storage servers

all you need to do is this..

pascalandy [8:18 PM]
OMG!
You guys are doing this as well?!
You saved the day!

The missing part - Install the volume driver

https://github.com/minio/minfs

docker plugin install minio/minfs

docker volume create

docker volume create -d minio/minfs \
--name bucket-dev-e \
-o endpoint=http://ip10_0_23_3-9001.play-with-docker.com/ \
-o access-key=A5a0a87b725552daXd \
-o secret-key=369f5e7b4a41e25452c353D629a24c372b62c90 \
-o bucket=bucket-dev-e

docker volume ls

Testing the volume within a container

docker run -d --name nginxtest1 -p 80:80 -v bucket-dev-e:/usr/share/nginx/html:ro nginx

Status 4)

By using our docker volume bucket-dev-e we can mount the bucket into any container. Very good!

Using sub directories from a bucket.

This part is work in progress. See https://github.com/minio/minfs/issues/20

For all details about my setup, please check my post:
The complete guide to attach a Docker volume with Minio on your Docker Swarm Cluster

— — —

Let’s say that my Minio's bucket is named: bucket-dev-e.
I mounted it here /mnt/minio00000/dev-e using docker volume create …

Let's start one blog (This works perfectly):

docker run --name some-ghost -v bucket-dev-e:/var/lib/ghost/content/images ghost

What if I need to run multiple websites:

docker run --name some-ghost -v bucket-dev-e/ghost/site1/images:/var/lib/ghost/content/images ghost

docker run --name some-ghost -v bucket-dev-e/ghost/site2/images:/var/lib/ghost/content/images ghost

My challange is … the commands above are not working. By default we cannot specify subpaths bucket-dev-e/ghost/site2/images from a Docker Volume.
What can we do ? (I DON’T KNOW THE ANSWER YET)

I don't want to use one Docker Volume for each of the 100x (potentially 1000x) site I’m hosting.

Any other ideas?

Conclusion

By using Minio along their minfs (https://github.com/minio/minfs) we can have best of both worlds.
A solid object storage and connect Docker volume to this storage. Any container can have access to the bucket created in Minio.

Another great thing with Minio is that you don't have to pre-define disk space (like GlusterFS, Infinit, Portworx, etc). Minio use whatever space you have a disk.

You can also create another data store easily on hyper.sh and rock to the world. It's been a long journey and now this will help me to move to production.

Cheers!
Pascal Andy | Twitter

[

Don't be shy to buzz me 👋 on Twitter @askpascalandy. Cheers!

The question is: What is the best practices for getting code into a container (git clone vs. copy vs. data container)

My answer: I use wget on the Github repo (branch master). Hope it helps!

FROM alpine:3.5


##############################################################################
# Install App
##############################################################################


WORKDIR $APP

RUN apk update && \
	apk upgrade && \
	apk --no-cache add tar curl tini \
    && apk --no-cache add --virtual devs gcc make python wget unzip ca-certificates \
	&& apk del devs gcc make python wget unzip ca-certificates \
	&& npm cache clean \
	&& rm -rf /tmp/npm*


##############################################################################
# PART ONE
# Install/copy FirePress_Klimax into casper from Github
##############################################################################
echo; echo; echo; \
echo "PART TWO ..."; echo; \

THEME_NAME_FROM="FirePress_Klimax"; \
THEME_NAME_INTO="casper"; \

GIT_URL="https://github.com/firepress-org/$THEME_NAME_FROM/archive/master.zip"; \

DIR_FROM="$DIR_THEMES/$THEME_NAME_FROM"; \
DIR_INTO="$DIR_THEMES/$THEME_NAME_INTO"; \

cd $DIR_THEMES; \
wget --no-check-certificate -O master.zip $GIT_URL; \
echo; echo; echo "List (12) $DIR_THEMES ..."; echo; ls -AlhF $DIR_THEMES; du -sh; echo; \

unzip $DIR_THEMES/master.zip; \
echo; echo; echo "List (13) $DIR_THEMES ..."; echo; ls -AlhF $DIR_THEMES; du -sh; echo; \

rm $DIR_THEMES/master.zip; \
echo; echo; echo "List (14) $DIR_THEMES ..."; echo; ls -AlhF $DIR_THEMES; du -sh; echo; \

mv $THEME_NAME_FROM-master $THEME_NAME_INTO; \
echo; echo; echo "List (15) $DIR_THEMES ..."; echo; ls -AlhF $DIR_THEMES; du -sh; echo; \

cd $GHOST_SOURCE; \
echo; echo; echo "List (16) $DIR_INTO ..."; echo; ls -AlhF $DIR_INTO; du -sh; echo; \

echo; echo; echo "Show $THEME_NAME_FROM version (17) ($DIR_INTO)"; echo; \
cat $DIR_INTO/package.json | grep "version"; \


##############################################################################
# # PART TWO
# Install Theme: XYZ
##############################################################################

# ... future themes
# ...
# ...


##############################################################################
# Clean up
##############################################################################
rm -rf /var/cache/apk/*; \
apk del wget unzip ca-certificates; \
echo "End of /RUN"; echo; echo; echo;à

[

P.S. If you have solid skills 🤓 with Docker Swarm, Linux and the things mention here and you would love 💚 to help a startup to launch 🔥 a solid project, I would love to get to know you 🍻. Buzz me 👋 on Twitter @askpascalandy. You can see the things that are done and the things we have to do here.

If you are curious about how things work like I am, I bet you will enjoy this post. I'll share the core elements that make FirePress an actual product you can use to grow your brand on the Web.

If you are not sure what FirePress is at this point, please check out our About section first.

Overall, we can easily deploy new clusters thanks to the way we design our backend. We even aim to completely wipe out our production(s) cluster(s) with a new one every 3-4 months.

We are cloud agnostic and there is no vendor lock-in at FirePress.

Services in production (application deployed via containers)

• Ghost is the core application that clients use to manage their websites (Node.js on the backend)
• Caddy serves landing pages
• Traefik proxy
• Letencrypt
• Portainer
• Resilio
• rClone
• EKL
• Prometheus
• Grafana

DevOps | High level

• Digital Ocean, Linux servers on Ubuntu 16.xx
• Cloud vendor (undisclosed at the moment)
• Micro services architecture
• Docker Swarm Orchestrator (Moby project)
• Well, you already understand that we deploy things in containers at this point.
• Container OS: Alpine (mainly)
• Bash-script wrapping around Docker commands
• Python
• Cloudflare
• Backblaze B2 (Object store)

Our official account on GitHub is https://github.com/firepress-org/