Encrypted, Tiered Backups: Building a Disaster Recovery System I Actually Trusted

A backup you've never restored isn't a backup

I've believed this for a long time, but building FlipLytics is where it stopped being a principle and became something I actually engineered around. A backup file sitting untouched in storage is a hypothesis. It's only actually a backup once you've proven, for real, that it can bring a system back from nothing.

What the system needed to do

FlipLytics' entire business depended on its PostgreSQL database staying intact. Losing it wouldn't just be an inconvenience, it would mean real sellers losing real profitability data they depended on. So the backup system needed to satisfy a few non-negotiable requirements: the data had to be encrypted, both in transit and at rest, it had to be genuinely restorable, not just theoretically restorable, and it had to run somewhere I trusted with direct database access.

Encryption, done properly

Backups were encrypted using AES-GCM, a mode that gives you both encryption and built-in integrity verification, rather than just encrypting bytes and hoping nothing got corrupted along the way. For anything beyond a small test database, encrypting the entire backup in memory in one pass isn't realistic, so the pipeline streams the backup in chunks, encrypting as it goes, rather than holding the entire thing in memory at once.

Tiered retention, not just "keep everything"

Keeping every single backup forever isn't a strategy, it's just deferred cleanup. I built a three-tier retention model instead: daily backups kept for a short window, weekly backups kept longer, and monthly backups kept longer still. Each tier automatically expires on its own schedule, so storage cost stays predictable and I always have appropriately-spaced recovery points without manually managing any of it.

Keeping database access away from public infrastructure

This is the part I'm most deliberate about. The backup and restore workflows run on self-hosted GitHub Actions runners, living inside my own Kubernetes cluster, specifically so that credentials with direct database access never have to touch GitHub's public runner infrastructure at all. It's a small architectural decision with a real security payoff: the blast radius of anything going wrong with a public runner simply doesn't include my production database.

Restore automation, and actually using it

The part most backup systems skip is automating the restore path at all, let alone testing it. I built full restore automation using pg_restore, driven by the same infrastructure as the backup jobs themselves, and I made a habit of actually running it, not just trusting that it would work when the day came that I genuinely needed it.

I also wired backup status into Discord, with proper structured notifications rather than a bare pass or fail. If a backup failed, I wanted to know immediately, with enough detail to actually act on it, not discover the gap days later when it was too late to matter.

Why this mattered more than it might seem

None of this is glamorous work. Nobody demos a backup system. But it's exactly the kind of infrastructure that only gets noticed when it's missing, at the worst possible moment. Building it properly, encrypted, tiered, genuinely tested, and isolated from unnecessary exposure, meant that a category of catastrophic failure simply wasn't something I had to worry about while building everything else.

That's the real value of good infrastructure: not that it does something impressive, but that it quietly removes an entire class of risk from your list of things to worry about.