ITSquad/Mastodon/Maintenance - Revision history

HgO: add pgbackrest and borg, remove autofs, and add new steps to upgrade Mastodon

2021-01-09T12:38:25Z

add pgbackrest and borg, remove autofs, and add new steps to upgrade Mastodon

HgO: /* Storage Box */

2019-09-03T20:25:14Z

Storage Box

← Older revision		Revision as of 22:25, 3 September 2019
Line 9:		Line 9:
	First, we need to create a ssh key pair for root and install it on the storage box:		First, we need to create a ssh key pair for root and install it on the storage box:
	ssh-keygen -t ed25519 -f /root/.ssh/storage-box		ssh-keygen -t ed25519 -f /root/.ssh/storage-box
	scp -P 23 .ssh/storage-box.pub <user>@<storage box server>:/home/.ssh		scp -P 23 .ssh/storage-box.pub <user>@<storage box server>:/home/.ssh/authorized_keys

	Then, we create a map for autofs in /etc/auto.master:		Then, we create a map for autofs in /etc/auto.master:

HgO: /* Storage Box */

2019-09-03T20:23:54Z

Storage Box

← Older revision		Revision as of 22:23, 3 September 2019
Line 7:		Line 7:
	This storage box is mounted with Autofs and sshfs.		This storage box is mounted with Autofs and sshfs.

	First, we create a map for autofs in /etc/auto.master:		First, we need to create a ssh key pair for root and install it on the storage box:
			ssh-keygen -t ed25519 -f /root/.ssh/storage-box
			scp -P 23 .ssh/storage-box.pub <user>@<storage box server>:/home/.ssh

			Then, we create a map for autofs in /etc/auto.master:
	/mnt/storage-box /etc/auto.sshfs uid=991,gid=991,--timeout=30,--ghost		/mnt/storage-box /etc/auto.sshfs uid=991,gid=991,--timeout=30,--ghost
	This line tells autofs to mount the content of /etc/auto.sshfs for the user with ID 991<ref>This is needed to avoid HTTP error 500 on file uploads. Source: https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Docker-Guide.md#using-a-prebuilt-image</ref>. After 30s, the mountpoint is disconnected if there isn't any activity. The ghost option will automatically create the mountpoints for us.		This line tells autofs to mount the content of /etc/auto.sshfs for the user with ID 991<ref>This is needed to avoid HTTP error 500 on file uploads. Source: https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Docker-Guide.md#using-a-prebuilt-image</ref>. After 30s, the mountpoint is disconnected if there isn't any activity. The ghost option will automatically create the mountpoints for us.

HgO at 20:13, 3 September 2019

2019-09-03T20:13:44Z

@@ Line 1: / Line 1: @@
 This page aims to describe procedures to maintain the [[ITSquad/Mastodon|Pirate Party's Mastodon]] instance.
 == How to upgrade? ==
@@ Line 56: / Line 78: @@
 In short, the option ''-Fc'' enables data compression. Postgres ensures data consistency during the backup<ref>https://www.postgresql.org/docs/current/app-pgdump.html</ref>, which means that we don't have to stop Mastodon while dumping the database ;)
-'''Important note''': At the moment, we are using this solution. Although it's not optimal (see below), it's the easiest solution to set up. Every night, a database dump is created. Database dumps are encrypted with GPG keys and stored on at least two different locations.
+'''Important note''': At the moment, we are using this solution. Although it's not optimal (see below), it's the easiest solution to set up. Every night, a database dump is created. These database dumps are encrypted with GPG keys and stored on at least two different locations.
 ==== WAL archiving ====

HgO: /* How to backup? */ remove duplicity commands from media section

2019-09-03T19:55:16Z

How to backup?: remove duplicity commands from media section

← Older revision		Revision as of 21:55, 3 September 2019
Line 114:		Line 114:
	This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set it at 14 days). Note that this command is daily executed on our instance.		This will remove local cache of media older than NUM_DAYS (=7 by default, but here we set it at 14 days). Note that this command is daily executed on our instance.

			'''Important note''': Due to the size of the media, we decided to store them on a storage box (500Go) provided by Hetzner. Every night, a snapshot of this storage box is scheduled, and we keep 4 snapshots.
	We can backup the medias using duplicity:<ref>http://duplicity.nongnu.org/</ref><ref>https://splone.com/blog/2015/7/13/encrypted-backups-using-rsync-and-duplicity-with-gpg-and-ssh-on-linux-bsd/</ref><ref>https://blog.rom1v.com/2013/08/duplicity-des-backups-incrementaux-chiffres/</ref>
	~~duplicity --encrypt-key <gpg encrypt key> \~~
	~~--full-if-older-than 1W \~~
	~~--name mastodon_medias \~~
	~~--num-retries 3 \~~
	~~--no-compression \~~
	~~public/system <protocol>://<backup server>:/path/to/backups~~
	This will store the encrypted backup on a remote server. The backups are incremental, but each week a full backup is made. Everything is encrypted with the provided GPG public keys. Duplicity needs to know at least one private key in order to do the incremental backups. The list of protocols that are handled by duplicity can be found in the man page. Finally, note that we don't compress the medias, since they are already compressed by dedicated video and image algorithms.


	~~As medias are really space costly, we don't keep more than one full backup (with all the daily incremental backups) at a time:~~
	~~duplicity remove-all-but-n-full 1 --force --name mastodon_medias <protocol>://<backup server>:/path/to/backups~~
	~~This command must be executed ''after'' each backup.~~

	'''Important note''': Due to the size of the media~~, we don't actually use this solution. Instead~~, we decided to store ~~the medias~~ on a storage box provided by Hetzner. Every night, a snapshot of this storage box is scheduled.

	=== Configuration files ===		=== Configuration files ===

	We ~~do the same as the medias, although we~~ can ~~space out~~ the ~~full~~ backups~~, as the files are modified less often~~:		We can backup the config files with duplicity:<ref>http://duplicity.nongnu.org/</ref><ref>https://splone.com/blog/2015/7/13/encrypted-backups-using-rsync-and-duplicity-with-gpg-and-ssh-on-linux-bsd/</ref><ref>https://blog.rom1v.com/2013/08/duplicity-des-backups-incrementaux-chiffres/</ref>
	duplicity --encrypt-key <gpg encrypt key> \		duplicity --encrypt-key <gpg encrypt key> \
	--full-if-older-than 1M \		--full-if-older-than 1M \
Line 143:		Line 128:
	--exclude '**' \		--exclude '**' \
	/ <protocol>://<backup server>:/path/to/backups		/ <protocol>://<backup server>:/path/to/backups

			This will store the encrypted backup on a remote server. The backups are incremental, but each week a full backup is made.

			Everything is encrypted with the provided GPG public keys. Duplicity needs to know at least one private key in order to do the incremental backups, so we create a key pair on the host.

			The list of protocols that are handled by duplicity can be found in the man page.

	In this example, we exclude everything except the /etc directory and the mastodon's config files.		In this example, we exclude everything except the /etc directory and the mastodon's config files.
Line 148:		Line 139:
	We can keep several full backups of the config files, as they weight almost nothing:		We can keep several full backups of the config files, as they weight almost nothing:
	duplicity remove-all-but-n-full 2 --force --name mastodon_config <protocol>://<backup server>:/path/to/backups		duplicity remove-all-but-n-full 2 --force --name mastodon_config <protocol>://<backup server>:/path/to/backups
			This command must be executed ''after'' each backup.

	'''Important note''': These two commands are executed every night, and backups are stored on at least two locations.		'''Important note''': These two commands are executed every night, and backups are stored on at least two locations.

HgO: /* How to upgrade? */

2019-09-02T19:45:40Z

How to upgrade?

← Older revision		Revision as of 21:45, 2 September 2019
Line 9:		Line 9:
	# git fetch -t # update the git branch, including new tags		# git fetch -t # update the git branch, including new tags
	# git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)		# git stash # prevent changes made to the files to be overwritten (mainly, the docker-compose.yml file)
	# git checkout glitch-soc/master # jump to the ~~version where we want to update~~		# git checkout glitch-soc/master # jump to the glitch-soc master
	# git stash pop # restore your changes		# git stash pop # restore your changes
	# docker-compose build # ~~pull from~~ the ~~docker-hub https://hub.docker.com/r/tootsuite/mastodon/~~		# docker-compose build # build the image
	# docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database		# docker-compose exec db pg_dump -U postgres -Fc postgres > ../dump_$(date +%d-%m-%Y"_"%H_%M_%S).sql # backup the database
	# tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.		# tail ../your_dump_file.sql # check if the backup worked, with your_dump_file.sql being the dumpfile you have created in the previous step.

HgO: /* How to upgrade? */

2019-09-02T19:44:35Z

How to upgrade?

← Older revision		Revision as of 21:44, 2 September 2019
Line 3:		Line 3:
	== How to upgrade? ==		== How to upgrade? ==

	A script called '''~~update~~.sh''' located in /home/mastodon will reproduce the steps below. Note: this script stops at step 11 and does not go further.		A script called '''upgrade.sh''' located in /home/mastodon will reproduce the steps below. Note: this script stops at step 11 and does not go further.

	Before running this script or the following commands, please check [https://github.com/tootsuite/mastodon/releases the instruction for the current release]. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)		Before running this script or the following commands, please check [https://github.com/tootsuite/mastodon/releases the instruction for the current release]. Sometimes, aditionnal actions are needed (other than migrate database and compile assets)

HgO: /* How to backup? */

2019-09-02T19:36:34Z

How to backup?

← Older revision		Revision as of 21:36, 2 September 2019
Line 47:		Line 47:
	== How to backup? ==		== How to backup? ==

	'''Currently''': Every night, an encrypted backup of the database is made on the server (~~but~~ is overwritten by the next nightly backup) and stored on at least two other locations. In addition, media are stored on a storage box provided by Hetzner. A snapshot of this storage box is scheduled every night. Finally, backups for configuration files are created every night. Those backups are encrypted and stored on at least two different locations.		'''Currently''': Every night, an encrypted backup of the database is made on the server (and is overwritten by the next nightly backup) and stored on at least two other locations. In addition, media are stored on a storage box provided by Hetzner. A snapshot of this storage box is scheduled every night. Finally, backups for configuration files are created every night. Those backups are encrypted and stored on at least two different locations.

	=== Database ===		=== Database ===

HgO: /* How to backup? */

2019-09-02T19:36:16Z

How to backup?

← Older revision		Revision as of 21:36, 2 September 2019
Line 47:		Line 47:
	== How to backup? ==		== How to backup? ==

	'''Currently''': Every night, a backup of the database is made on the server (but is overwritten by the next nightly backup). ~~This is clearly not optimal~~. ~~Below~~ is ~~the ideal situation we would like to reach~~. ~~We '''do not''' backup medias and~~ configuration files ~~yet~~. ~~'''We still need to implement a more reliable solution'''~~		'''Currently''': Every night, an encrypted backup of the database is made on the server (but is overwritten by the next nightly backup) and stored on at least two other locations. In addition, media are stored on a storage box provided by Hetzner. A snapshot of this storage box is scheduled every night. Finally, backups for configuration files are created every night. Those backups are encrypted and stored on at least two different locations.

	=== Database ===		=== Database ===
Line 129:		Line 129:
	This command must be executed ''after'' each backup.		This command must be executed ''after'' each backup.

	'''Important note''': Due to the size of the media, we don't actually use this solution. Instead, we decided to store the medias on a storage box provided by Hetzner. Every ~~day~~, a snapshot of this storage box is scheduled.		'''Important note''': Due to the size of the media, we don't actually use this solution. Instead, we decided to store the medias on a storage box provided by Hetzner. Every night, a snapshot of this storage box is scheduled.

	=== Configuration files ===		=== Configuration files ===
Line 149:		Line 149:
	duplicity remove-all-but-n-full 2 --force --name mastodon_config <protocol>://<backup server>:/path/to/backups		duplicity remove-all-but-n-full 2 --force --name mastodon_config <protocol>://<backup server>:/path/to/backups

	'''Important note''': These two commands are executed every night, and backups are stored to at least two locations.		'''Important note''': These two commands are executed every night, and backups are stored on at least two locations.

	== References ==		== References ==

	<references/>		<references/>

HgO: /* Database */

2019-09-02T19:33:07Z

Database

← Older revision		Revision as of 21:33, 2 September 2019
Line 55:		Line 55:

	In short, the option ''-Fc'' enables data compression. Postgres ensures data consistency during the backup<ref>https://www.postgresql.org/docs/current/app-pgdump.html</ref>, which means that we don't have to stop Mastodon while dumping the database ;)		In short, the option ''-Fc'' enables data compression. Postgres ensures data consistency during the backup<ref>https://www.postgresql.org/docs/current/app-pgdump.html</ref>, which means that we don't have to stop Mastodon while dumping the database ;)

			'''Important note''': At the moment, we are using this solution. Although it's not optimal (see below), it's the easiest solution to set up. Every night, a database dump is created. Database dumps are encrypted with GPG keys and stored on at least two different locations.

	==== WAL archiving ====		==== WAL archiving ====