ITSquad/Mastodon/Maintenance

This page aims to describe procedures to maintain the Pirate Party's Mastodon instance.

Storage Box

Due to the size of the media, we decided to store them on a storage box provided by Hetzner. This can also be used for storing our backups.

This storage box is mounted with sshfs.

First, we need to create a ssh key pair for the root user and install it on the storage box. In the ITSquad, we decided to create a sub-account of the storage-box per server, so we can safely overwrite the storage box's authorized_keys file:

ssh-keygen -t ed25519 -f /root/.ssh/storage-box
scp -P 23 .ssh/storage-box.pub <user>@<storage box server>:/home/.ssh/authorized_keys

Then, we create the directory that will be used as the storage box's mountpoint:

mkdir /mnt/mastodon

We add the following line in /etc/fstab to mount the storage box volume:

# <file system>                     <mount point>  <type>      <options>  <dump>  <pass>
<storage box>:/home/media  /mnt/mastodon  fuse.sshfs delay_connect,_netdev,user,idmap=user,transform_symlinks,allow_other,default_permissions,reconnect,uid=mastodon,gid=mastodon 0 0

We give the ownership of this volume to the mastodon user in order to avoid HTTP error 500 on file uploads..

We configure the SSH client config at /root/.ssh/config to define which port, user and private key to use to connect to the storage box:

Host <storage box>
  User <user>
  Port 23
IdentityFile /root/.ssh/storage-box
PreferredAuthentications publickey,password

We mount the storage box volume:

mount /mnt/mastodon

Finally, we create a link on the mastodon directory:

ln -s /mnt/mastodon/media /path/to/mastodon/public/system

And we restart the Mastodon instance \o/

Note: We first tried to use autofs to mount the storage box, but this was resulting in socket errors because the docker container didn't get the information that the volume had been disconnected and mounted again.

How to upgrade?

A script called upgrade.sh located in /home/mastodon will reproduce the steps below. Note: this script stops at step 12 and does not go further.

Before running this script or the following commands, please check the instruction for the current release. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)

1. git fetch -t # update the git branch, including new tags
2. git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
3. git pull
4. git stash pop # restore your changes
5. docker-compose build # build the image
6. docker-compose exec -u postgres pgbackrest --retention-diff 3 --stanza main --type incr backup # backup the database (see #WAL_archiving with_pgBackRest)
7. tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.
8. docker-compose run --rm -e SKIP_POST_DEPLOYMENT_MIGRATIONS=true web rails db:migrate # upgrade the database (pre-install)
9. docker-compose down # shut down the containers.
10. docker-compose run --rm web bin/tootctl cache clear # clear the cache
11. docker-compose run --rm web rails db:migrate # upgrade the database
12. docker-compose up -d # start the mastodon instance (create new volumes)
13. docker-compose logs -ft web # (optional) if you want to monitor the progress. Once this is done you ctrl+c
14. docker system prune -a # remove all unused volumes, old images, etc.

What to do when something went wrong?

Don't panic.

If you made a dump of the database before upgrading, you can restore the database as follows:

1. docker-compose stop
2. docker-compose start db
3. docker-compose exec db dropdb postgres -U postgres # remove the db !!!!!!!
4. docker-compose exec db createdb postgres -U postgres # create a fresh and new db
5. cat ../your_dump_file.sql | docker exec -i mastodon_db_1 psql -U postgres # restore the database, with "your_dump_file" being a database backup
6. docker-compose down
7. docker-compose up -d

If you are using pgBackRest, you can follow these steps: https://pgbackrest.org/user-guide.html#quickstart/perform-restore

As the tool is inside a docker container, you can enter the container like this:

docker-compose run --rm -u postgres db bash

Then, you can execute the commands from there.

What to backup?

Backups of the database are not enough. We need to backup medias, user feeds, etc.

According to the official Mastodon documentation, we need to take special cares of the following files and directories:

• The public/system directory, which contains user-uploaded images and videos
• The .env.production and docker-compose.yml files, which contain server config and secrets
• The Postgres database, using pg_dump (see below)
• The /etc directory, which contains the system's configuration files

Moreover, the backups must be encrypted on the storage server.

How to backup?

Currently: Every night, an encrypted backup of the database is made on the server and is stored on at least two other locations. The backups are incremental every day, differential every Sunday, and we perform a full backup every month. Backups for configuration files are created every night. These backups are incremental and are stored for 6 months and are stored on at least two different locations. Finally, backups and media are stored on a storage box provided by Hetzner. A snapshot of this storage box is scheduled every night.

Database

For an instant backup, we can just dump the database as follows:

docker-compose exec -u postgres db pg_dump -Fc postgres > /home/mastodon/backup/db/dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql

In short, the option -Fc enables data compression. Postgres ensures data consistency during the backup, which means that we don't have to stop Mastodon while dumping the database ;)

WAL archiving with pgBackRest

However, over a long time period, we usually wants incremental backups to avoid duplicating the whole database each day. Fortunately, Postgres provides all the tools we need to set up incremental backups.

First, we have to enable WAL (Write Ahead Log) archiving.. We just modify the Postgres config, which is located in /path/to/mastodon/postgres/postgressql.conf:

# The WAL level must be archive or higher.
wal_level = replica

# This is a soft upper limit on the total size of WAL files.
max_wal_size = 1GB

# The archive_mode must be set to on for archiving to happen.
archive_mode = on

# This is the command to invoke for each WAL file to be archived.
archive_command = '/usr/local/bin/pgbackrest --log-level-console=info --stanza=main archive-push %p'

# Max number of processes to generate the WAL files
max_wal_senders=3

This configuration tells PostgreSQL to archive transaction by executing the command:

/usr/local/bin/pgbackrest --log-level-console=info --stanza=main archive-push %p

Note that this command must be executable by the postgres user. If the script fails, PostgreSQL will try to execute it again and again. Please check the Postgres logs to be sure that everything is fine.

Then, we must install pgBackRest, which is the tool that will backup the WAL files. Although the user guide of pgBackRest is dense, it is quite straightforward, so we recommend you to read it.

Since we are using docker, we had to create a custom docker image that includes pgBackRest. We created the following Dockerfile in /home/mastodon/pgbackrest/Dockerfile:

FROM postgres:12-alpine

ARG PGBACKREST_VERSION="2.31"
ARG PGBACKREST_SHA256SUM="7157ec4ad2428379243c30acf2b15c2e9339beeec14697714f8eac2ce4c19896"
RUN apk add --virtual build-dependencies \
      build-base gcc make wget postgresql-dev openssl-dev libxml2-dev pkgconfig lz4-dev bzip2-dev \
    && apk add lz4-libs coreutils libbz2 \
    && pgbackrest_archive="pgbackrest-${PGBACKREST_VERSION}.tar.gz" \
    && pgbackrest_dir="/opt/pgbackrest-release-${PGBACKREST_VERSION}" \
    && wget -O $pgbackrest_archive -q https://github.com/pgbackrest/pgbackrest/archive/release/${PGBACKREST_VERSION}.tar.gz \
    && echo "${PGBACKREST_SHA256SUM}  $pgbackrest_archive" | sha256sum -c - \
    && tar zxf $pgbackrest_archive -C /opt \
    && cd $pgbackrest_dir/src \
    && ./configure \
    && make \
    && make install \
    && ln -s /usr/local/bin/pgbackrest /usr/bin/pgbackrest \
    && apk del build-dependencies \
    && cd / \
    && rm -rf $pgbackrest_archive $pgbackrest_dir

COPY --chown=postgres:postgres pgbackrest.sh /opt/
RUN chmod 744 /opt/pgbackrest.sh

COPY --chown=postgres:postgres pgbackrest.conf /etc/
RUN chmod 640 /etc/pgbackrest.conf

CMD ["postgres", \ 
      "-c", "archive_command=/usr/local/bin/pgbackrest --log-level-console=info --stanza=main archive-push %p", \
      "-c", "archive_mode=on", \
      "-c", "wal_level=replica", \
      "-c", "max_wal_senders=3"]

Because pgbackrest wasn't packaged for alpine images, we had to build it from sources.

We also create the pgBackRest config file in /home/mastodon/pgbackrest/pgbackrest.conf:

[global]
log-level-file=off
log-level-console=info

repo1-path=/mnt/pgbackrest
repo1-retention-full=2
repo1-retention-diff=2
repo1-cipher-pass=<pgbackrest password>
repo1-cipher-type=aes-256-cbc

[main]
pg1-path=/var/lib/postgresql/data

Don't forget to generate your pgbackrest password.

Finally, change the /path/to/mastodon/docker-compose.yml file to build and use this custom postgres docker image:

db:
  ...
  build: /home/mastodon/pgbackrest
  image: pgbackrest:12
  ...
  volumes:
    ...
    - /mnt/pgbackrest:/mnt/pgbackrest
  ...

Here, we can see that we are mounting a volume that will contain the backup. We use the same method as the one for storing the media (see #Storage_Box), and we gave the ownership to the postgres user, i.e. the uid 70 as this user only exists inside the postgres docker container.

Note: We upgraded the PostgreSQL database from 9.6 to 12, so adapt this configuration according to your needs.

Finally, build the docker image and recreate the database container:

docker-compose build db
docker-compose down
docker-compose up -d

Medias

It can be interesting to execute the following command before making a backup of the medias:

docker-compose run --rm web bin/tootctl media remove --days=14

This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set it at 14 days). Note that this command is daily executed on our instance.

Important note: Due to the size of the media, we decided to store them on a storage box (500Go) provided by Hetzner. Every night, a snapshot of this storage box is scheduled, and we keep 4 snapshots.

Configuration files

We are backuping the config files with Borg and borgmatic.

We created a storage box mountpoint at /mnt/backup (see #Storage_Box) and we gave ownership to the root user and the backup-sync group

First, we initialize the Borg repository:

borg init -e keyfile --umask 0027 /mnt/backup/borg

This will prompt for a passphrase. You'll have to backup the keyfile as it is needed to decrypt the borg repository, and it is stored on the server, under /root/.config/borg/keys

The umask option is there to allow read-write permission to the owner, allow read-only permission to the group, and remove all permissions for other users. See also the Wikipedia about umask.

Then, we create the borgmatic config file at /etc/borgmatic/config.yaml:

location:
  exclude_patterns:
  - /path/to/elasticsearch
  - /path/to/redis
  repositories:
  - <storage box>:/home/backup/borg
  source_directories:
  - /path/to/mastodon
  - /etc
retention:
  keep_daily: 7
  keep_hourly: 24
  keep_monthly: 6
  keep_weekly: 4
storage:
  compression: zlib,7
  encryption_passphrase: <borg passphrase>
  umask: '0027'

We can now trigger a backup with this command:

borgmatic --create --prune

We can also check the consistency of the backup with:

borgmatic --check

Important note: These two commands are executed every night, and backups are stored on at least two locations.