Reason for month-long downtime

If you recall, I broke my site really badly in January, and it was down for a month till February.

The problem

I said I’d explain what happened; instead of repeating myself, I’ll share the long and detailed report that I sent my hosting provider, Digital Ocean (DO):

The first thing I did was to look on the internet for a solution. As explained in my report, I followed the advice in the output of each command, but to no avail. The internet had a couple of answers [1,2], but neither helped. That’s when I turned to DO for help.

Digital Ocean’s reply

DO wrote me a lovely email telling me I’d done goofed so badly (not in those words) that the easiest thing to do would be to spin up a new droplet, transfer all content from the broken one to the new one, then delete the broken one. And make backups and snapshots as I go. DO couldn’t emphasise making snapshots enough during our correspondence.

The solution

The fix seemed simple enough, but there were a lot of hoops to jump through. I took snapshots before I jumped through the trickier hoops. As it turned out, I had to spin up a second new droplet because I messed up the first one. I think that’s when I finally accepted I’d be better off paying for backups and snapshots. Spending the money is better than the time taken and stress caused by breaking everything.

It would take forever to tell you every last detail, so, instead of telling you every last detail, I’ve listed some of the problems I faced here:

• configuring Apache, especially with respect to forwarding from HTTP to HTTPS [3,4,5] and from www.pandammonium.org to pandammonium.org.
• working out whether any DNS settings in DreamHost and CloudFlare needed changing
• installing and configuring WordPress
• correcting file permissions
• correcting file ownership
• getting the MySQL content from the broken droplet and putting it in the new one [6,7]
• becoming familiar enough with rsync so as not to delete the contents of my local hard drive (thank goodness I’d used the backup option) [8]
• understanding and configuring a plugin called Fail2ban that DO installed [9,10,11]
• configuring OpenSSH [12]
• finding where local executables should be placed in Ubuntu [13]

It was all very exhausting and stressful at the time, which is why I didn’t write about it then. Thankfully, all the details and stresses are floating away now, which is one reason why I’m not going into detail now. (The other reason is that it would take forever to write, and would result in a really long post.)

Incident report

I’ve added a backfilled incident report on my new status page.

References

1. https://askubuntu.com/questions/298177/a-failed-to-fetch-error-occurs-when-apt-get-update-is-run-how-do-i-fix-this/298242#298242
2. https://askubuntu.com/questions/298177/a-failed-to-fetch-error-occurs-when-apt-get-update-is-run-how-do-i-fix-this/477824#477824
3. https://linuxize.com/post/redirect-http-to-https-in-apache/
4. https://stackoverflow.com/questions/16200501/how-to-automatically-redirect-http-to-https-on-apache-servers
5. https://www.ssl.com/how-to/redirect-http-to-https-with-apache/
6. https://phoenixnap.com/kb/import-and-export-mysql-database
7. https://simplebackups.com/blog/the-complete-mysqldump-guide-with-examples/
8. https://www.howtogeek.com/135533/how-to-use-rsync-to-backup-your-data-on-linux/
9. https://www.fail2ban.org/wiki/index.php/Main_Page
10. https://linuxize.com/post/install-configure-fail2ban-on-ubuntu-20-04/
11. https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-20-04
12. https://www.ssh.com/academy/ssh/sshd_config
13. https://unix.stackexchange.com/questions/36871/where-should-a-local-executable-be-placed/36874#36874