Postmortem of the March 2, 2021 outage

On March 2, 2021, we had a large incident that led to the entire acoustid.org infrastructure going down. This will be the first of at least a few posts to explain what happened, how we got to this situation and what will happen in the future? I’ll not sugar-coat it, the most direct cause of the problem was the modern equivalent of the classic rm -rf / as root.