Moving 900+ GB of Prometheus Without a Cutover Weekend

The Prometheus I was responsible for had grown past 900 GB of time series data, and at that size it had every disease that comes with it. Queries timed out unless you narrowed them down to an hour. Cardinality kept growing, because teams added labels “for debugging” and then sometimes forgot to remove them.