Difference between revisions of "ITSquad/Mastodon/Maintenance"

From Pirate Party Belgium
Jump to: navigation, search
(How to backup?: we have backups)
m (How to backup?)
(11 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
== How to upgrade? ==
 
== How to upgrade? ==
  
A script called '''update.sh''' located in /home/mastodon will reproduce the steps below. Note: this script stops at step 14 and does not go further.
+
A script called '''update.sh''' located in /home/mastodon will reproduce the steps below. Note: this script stops at step 11 and does not go further.
  
 
Before running this script or the following commands, please check [https://github.com/tootsuite/mastodon/releases the instruction for the current release]. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)
 
Before running this script or the following commands, please check [https://github.com/tootsuite/mastodon/releases the instruction for the current release]. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)
  
# git fetch # update the git branch
+
# git fetch -t # update the git branch, including new tags
 
# git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
 
# git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
 
# git checkout vX.X.X # jump to the version where we want to update
 
# git checkout vX.X.X # jump to the version where we want to update
 
# git stash pop # restore your changes
 
# git stash pop # restore your changes
# docker-compose pull # pull from the docker-hub https://hub.docker.com/r/gargron/mastodon/
+
# docker-compose pull # pull from the docker-hub https://hub.docker.com/r/tootsuite/mastodon/
# docker-compose stop # stop the containers
 
# docker-compose start db # start the db container
 
 
# docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database
 
# docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database
 
# tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.
 
# tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.
 
# docker-compose down # shut down the containers.  
 
# docker-compose down # shut down the containers.  
#: You'll maybe get "ERROR: An HTTP request took too long to complete" and other errors. Don't mind this and just wait 'till it's done.
+
#: # You'll maybe get "ERROR: An HTTP request took too long to complete" and other errors. Don't mind this and just wait 'till it's done.
 
# docker-compose run --rm web rails db:migrate # upgrade the database
 
# docker-compose run --rm web rails db:migrate # upgrade the database
 
# docker-compose run --rm web rails assets:precompile # complie the assets
 
# docker-compose run --rm web rails assets:precompile # complie the assets
 
# docker-compose up -d # start the mastodon instance (create new volumes)
 
# docker-compose up -d # start the mastodon instance (create new volumes)
 
# docker-compose logs -ft web # (optional) if you want to monitor the progress. Once this is done you ctrl+c
 
# docker-compose logs -ft web # (optional) if you want to monitor the progress. Once this is done you ctrl+c
# docker volume prune # remove all unused volumes
+
# docker system prune # remove all unused volumes, images, etc.
  
 
=== What to do when something went wrong? ===
 
=== What to do when something went wrong? ===
Line 37: Line 35:
 
You can also go to a previous version of Mastodon with:
 
You can also go to a previous version of Mastodon with:
 
# git checkout vX.X.X
 
# git checkout vX.X.X
# sed s/:latest/:vX.X.X/g docker-compose.yml # don't forget to change this back to "latest" when things get fixed
+
# sed -r 's/(image:.+mastodon)(:.+)?$/\1:vX.X.X/g' docker-compose.yml # don't forget to change this back to "latest" when things get fixed
 
# docker-compose pull
 
# docker-compose pull
  
Line 48: Line 46:
  
 
Then, we backup the database:
 
Then, we backup the database:
  docker-compose exec db pg_dump -U postgres -Fc postgres > /home/mastodon/backup/db/dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
+
  docker-compose exec db pg_dump -U postgres -Fc postgres | bzip2 --best > /home/mastodon/backup/db/dump_$(date +%d-%m-%Y"_"%H_%M_%S).bz2
  
 
It can be interesting to execute the following command before doing a backup:
 
It can be interesting to execute the following command before doing a backup:
  NUM_DAYS=14 docker-compose run --rm web rails mastodon:media:remove_remote
+
  docker-compose run --rm web bin/tootctl media remove --days=14
 
This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set at 14 days). Note that this command is executed every day on our instance.
 
This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set at 14 days). Note that this command is executed every day on our instance.
  
  
  
Currently, [[User:HgO|HgO]] is doing a daily backup (db + media), and keep them up to 7 days. Data are stored on his laptop.  
+
<s>Currently, [[User:HgO|HgO]] is doing a daily backup (db + media), and keep them up to 7 days. Data are stored on his laptop.</s> We need a backup solution…
  
Every night, a backup of the db is also made on the server (but is overwritten by the nightly backup).
+
Every night, a backup of the db is also made on the server (but is overwritten by the next nightly backup).
  
 
'''We still need to implement a more reliable solution'''
 
'''We still need to implement a more reliable solution'''

Revision as of 22:19, 26 December 2018

This page aims to describe procedures to maintain the Pirate Party's Mastodon instance.

How to upgrade?

A script called update.sh located in /home/mastodon will reproduce the steps below. Note: this script stops at step 11 and does not go further.

Before running this script or the following commands, please check the instruction for the current release. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)

  1. git fetch -t # update the git branch, including new tags
  2. git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
  3. git checkout vX.X.X # jump to the version where we want to update
  4. git stash pop # restore your changes
  5. docker-compose pull # pull from the docker-hub https://hub.docker.com/r/tootsuite/mastodon/
  6. docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database
  7. tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.
  8. docker-compose down # shut down the containers.
    # You'll maybe get "ERROR: An HTTP request took too long to complete" and other errors. Don't mind this and just wait 'till it's done.
  9. docker-compose run --rm web rails db:migrate # upgrade the database
  10. docker-compose run --rm web rails assets:precompile # complie the assets
  11. docker-compose up -d # start the mastodon instance (create new volumes)
  12. docker-compose logs -ft web # (optional) if you want to monitor the progress. Once this is done you ctrl+c
  13. docker system prune # remove all unused volumes, images, etc.

What to do when something went wrong?

Don't panic. You can restore the database backup as follows:

  1. docker-compose stop
  2. docker-compose start db
  3. docker-compose exec db dropdb postgres -U postgres # remove the db !!!!!!!
  4. docker-compose exec db createdb postgres -U postgres # create a fresh and new db
  5. cat ../your_dump_file.sql | docker exec -i mastodon_db_1 psql -U postgres # restore the database, with "your_dump_file" being a database backup
  6. docker-compose down
  7. docker-compose up -d

You can also go to a previous version of Mastodon with:

  1. git checkout vX.X.X
  2. sed -r 's/(image:.+mastodon)(:.+)?$/\1:vX.X.X/g' docker-compose.yml # don't forget to change this back to "latest" when things get fixed
  3. docker-compose pull

How to backup?

Backups of the database are not enough. We need to backup images, user feeds, etc.

We can simply copy the whole Mastodon's directory. For instance:

tar cvf /home/mastodon/backup/mastodon.tar /home/mastodon/mastodon/

Then, we backup the database:

docker-compose exec db pg_dump -U postgres -Fc postgres | bzip2 --best > /home/mastodon/backup/db/dump_$(date +%d-%m-%Y"_"%H_%M_%S).bz2

It can be interesting to execute the following command before doing a backup:

docker-compose run --rm web bin/tootctl media remove --days=14

This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set at 14 days). Note that this command is executed every day on our instance.


Currently, HgO is doing a daily backup (db + media), and keep them up to 7 days. Data are stored on his laptop. We need a backup solution…

Every night, a backup of the db is also made on the server (but is overwritten by the next nightly backup).

We still need to implement a more reliable solution