ITSquad/Mastodon/Maintenance

This page aims to describe procedures to maintain the Pirate Party's Mastodon instance.

Storage Box

Due to the size of the media, we decided to store them on a storage box provided by Hetzner.

This storage box is mounted with Autofs and sshfs.

First, we need to create a ssh key pair for root and install it on the storage box:

ssh-keygen -t ed25519 -f /root/.ssh/storage-box
scp -P 23 .ssh/storage-box.pub <user>@<storage box server>:/home/.ssh/authorized_keys

Then, we create a map for autofs in /etc/auto.master:

/mnt/storage-box /etc/auto.sshfs uid=991,gid=991,--timeout=30,--ghost

This line tells autofs to mount the content of /etc/auto.sshfs for the user with ID 991. After 30s, the mountpoint is disconnected if there isn't any activity. The ghost option will automatically create the mountpoints for us.

The content of /etc/auto.sshfs is as follows:

media -fstype=fuse,rw,allow_other,port=23,_netdev :sshfs\#<user>@<storage box server>\:/home/media

This simply tells autofs to mount the media folder from the storage box server with sshfs on the directory /mnt/storage-box/media.

We restart the autofs service to make the changes effective:

sudo systemctl restart autofs

Finally, we create a link on the mastodon directory:

ln -s /mnt/storage-box/media /path/to/mastodon/public/system

And we restart the Mastodon instance \o/

How to upgrade?

A script called upgrade.sh located in /home/mastodon will reproduce the steps below. Note: this script stops at step 11 and does not go further.

Before running this script or the following commands, please check the instruction for the current release. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)

1. git fetch -t # update the git branch, including new tags
2. git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
3. git checkout glitch-soc/master # jump to the glitch-soc master
4. git stash pop # restore your changes
5. docker-compose build # build the image
6. docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database
7. tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.
8. docker-compose down # shut down the containers.

   # You'll maybe get "ERROR: An HTTP request took too long to complete" and other errors. Don't mind this and just wait 'till it's done.

9. docker-compose run --rm web rails db:migrate # upgrade the database
10. docker-compose run --rm web rails assets:precompile # complie the assets
11. docker-compose up -d # start the mastodon instance (create new volumes)
12. docker-compose logs -ft web # (optional) if you want to monitor the progress. Once this is done you ctrl+c
13. docker system prune -a # remove all unused volumes, old images, etc.

What to do when something went wrong?

Don't panic. You can restore the database backup as follows:

1. docker-compose stop
2. docker-compose start db
3. docker-compose exec db dropdb postgres -U postgres # remove the db !!!!!!!
4. docker-compose exec db createdb postgres -U postgres # create a fresh and new db
5. cat ../your_dump_file.sql | docker exec -i mastodon_db_1 psql -U postgres # restore the database, with "your_dump_file" being a database backup
6. docker-compose down
7. docker-compose up -d

What to backup?

Backups of the database are not enough. We need to backup medias, user feeds, etc.

According to the official Mastodon documentation, we need to take special cares of the following files and directories:

• The public/system directory, which contains user-uploaded images and videos
• The .env.production and docker-compose.yml files, which contain server config and secrets
• The Postgres database, using pg_dump (see below)
• The /etc directory, which contains the system's configuration files

Moreover, the backups must be encrypted on the storage server.

How to backup?

Currently: Every night, an encrypted backup of the database is made on the server (and is overwritten by the next nightly backup) and stored on at least two other locations. In addition, media are stored on a storage box provided by Hetzner. A snapshot of this storage box is scheduled every night. Finally, backups for configuration files are created every night. Those backups are encrypted and stored on at least two different locations.

Database

For an instant backup, we can just dump the database as follows:

docker-compose exec db pg_dump -U postgres -Fc postgres > /home/mastodon/backup/db/dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql

In short, the option -Fc enables data compression. Postgres ensures data consistency during the backup, which means that we don't have to stop Mastodon while dumping the database ;)

Important note: At the moment, we are using this solution. Although it's not optimal (see below), it's the easiest solution to set up. Every night, a database dump is created. These database dumps are encrypted with GPG keys and stored on at least two different locations.

WAL archiving

However, over a long time period, we usually wants incremental backups to avoid duplicating the whole database each day. Fortunately, Postgres provides all the tools we need to set up incremental backups.

First, we have to enable WAL (Write Ahead Log) archiving.. We just modify the Postgres config, which is located in /path/to/mastodon/postgres/postgressql.conf:

# The WAL level must be archive or higher.
wal_level = archive

# Ensure there is at least one WAL file for each "archive_timeout" duration.
archive_timeout = 1h

# This is a soft upper limit on the total size of WAL files.
max_wal_size = 1GB

# Keep around at least these many WAL files (aka segments).
wal_keep_segments = 10

# The archive_mode must be set to on for archiving to happen.
archive_mode = on

# This is the command to invoke for each WAL file to be archived.
archive_command = '/opt/wal-backup.sh wal %p'

This configuration tells Postgrel to archive transaction at least every hour by executing the script in /opt/wal-backup.sh. Note that this script must be executable by the postgres user. If the script fails, Postgres will try to execute it again and again. Please check the Postgres logs to be sure that everything is fine.

Then, we must install a tool that will backup the WAL files. We can either use WAL-E or WAL-G which is a refactor in Go of the former. Both tools' configuration are very similar.

For WAL-G, we would end up with the following script:

# If the script is run on the same host as Postgres, we can just listen to the Postgres' socket
export PGHOST=/var/run/postgresql
export PGUSER=postgres

# This variable expect a string with the public key in ascii
export WALG_PGP_KEY=$(cat /path/to/your-key.pub)

export WALG_FILE_PREFIX=/path/to/backups/postgres/

# Postgres doesn't export the $PATH variable, so we must set it up
export PATH=/usr/bin:/usr/local/bin:.

# The actual backup command. $1 is either "wal" or "backup", and $2 is the path to the file or database.
wal-g $1-push $2

You can export your public key to a file with:

gpg -a --export <gpg key id> > /path/to/your-key.pub

Note that WAL-G doesn't currently support GPG keys using elliptic curves.

Finally, we must create an initial base backup once:

/opt/wal-backup.sh backup /path/to/mastodon/postgres

Medias

It can be interesting to execute the following command before making a backup of the medias:

docker-compose run --rm web bin/tootctl media remove --days=14

This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set it at 14 days). Note that this command is daily executed on our instance.

Important note: Due to the size of the media, we decided to store them on a storage box (500Go) provided by Hetzner. Every night, a snapshot of this storage box is scheduled, and we keep 4 snapshots.

Configuration files

We can backup the config files with duplicity:

duplicity --encrypt-key <gpg encrypt key> \
  --full-if-older-than 1M \
  --name daily_mastodon_config_backup \
  --num-retries 3 \
  --include /etc \
  --include /path/to/mastodon/.env.production \
  --include /path/to/mastodon/docker-compose.yml \
  --exclude '**' \
  / <protocol>://<backup server>:/path/to/backups

This will store the encrypted backup on a remote server. The backups are incremental, but each week a full backup is made.

Everything is encrypted with the provided GPG public keys. Duplicity needs to know at least one private key in order to do the incremental backups, so we create a key pair on the host.

The list of protocols that are handled by duplicity can be found in the man page.

In this example, we exclude everything except the /etc directory and the mastodon's config files.

We can keep several full backups of the config files, as they weight almost nothing:

duplicity remove-all-but-n-full 2 --force --name mastodon_config <protocol>://<backup server>:/path/to/backups

This command must be executed after each backup.

Important note: These two commands are executed every night, and backups are stored on at least two locations.