Earlier this year I started consolidating some workloads to my homelab Kubernetes cluster. One of the last ones was a doozy. It's not a compute-hard or a memory-hard workload, it's a storage-hard workload. I needed to move the DJ set recording bot for an online radio station off of its current cloud and onto my homelab, but I still wanted the benefits of the cloud such as no thinking remote backups.
This bot has been running for a decade and the dataset well predates that, over 675 Gi of DJ sets, including ones that were thought to be lost media. Each of these sets is a 320 KiB/sec MP3 file that is anywhere from 150 to 500 MB, with smaller text files alongside them.
Needless to say, this dataset is very important to me. The community behind this radio station is how I've met some of my closest friends. I want to make sure that it's backed up and available for anyone that wants to go back and listen to the older sets. I want to preserve these files and not just dump them in an Archive bucket or something that would make it hard or slow to access them. I want these to be easily accessible to help preserve the work that goes into live music performances.
Here's how I did it and made it easy with Tigris.
An extreme close-up of a tiger with blue and orange fur. The Kubernetes logo replaces its iris.