Backups Aren't Simple Backups aren't simple Software Engineer | FOSS Enthusiast | Homelabber --> Backups aren't simple Aleksandar Filipovski, 2026-09-16 See also: John Salvatier’s excellent blog, Rea
By Coderz Club · 2026-09-17 · Tags: html
Backups Aren't Simple
Backups aren't simple Software Engineer | FOSS Enthusiast | Homelabber --> Backups aren't simple Aleksandar Filipovski, 2026-09-16 See also: John Salvatier’s excellent blog, Reality has a surprising amount of detail I read a comment somewhere that stuck with me, that went something like this: “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” Trying to find the source for it for this blog, it turned out that every other sysadmin has his rehashed version of the quote, but the gist of it is the same everywhere. Data loss is something that happens more often than we’d hope, and most of us are woefully unprepared for when it hits us (which is almost always at the worst possible time). I can confirm that I had a similar experience once. When I was little we had pulled all our family photos from our home laptops and PCs onto an external hard drive, in order to free up some space. This worked beautifully until one day my dad wanted to use the drive as storage for our TV set-top box (one of these old things), and was prompted to format the drive. He went ahead with it, and the disk was reformatted. The index of files was deleted, and we were stuck with a nominally empty drive. It would be easy to blame him for screwing up, but it takes beginning a career in tech to realise that there is a series of errors that lead to this kind of mistake. Firstly, we had put all of our photos in one place and didn’t bother with backups. Secondly, most consumer-facing software usually has bold disclaimers telling you that formatting a disk means losing data (which the set-top box didn’t, terrible UI). Besides, why would you even expect a non-technical person to even have to know any of this? Thankfully we were able to get the photos restored, and it turned out to be a cheap lesson in handling data. You never keep important things in one place only. There’s about a million things that can go wrong. Your drive could die, it could be stolen, bits could rot in cold storage (hard drives have magnetic particles which can inexplicably shift, and SSDs are made of NAND transistors which leak electricity and over time, corrupt your data). So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good. This doesn’t cover the headaches of what a plugged in drive could do. Ransomware could encrypt your files, and you could do anything from an honest mistake like deleting the wrong file; up to catastrophic mistakes like running a script that overwrites everything with zeroes. So our backup should not be a mirror of the first drive, because we also want to be able to go back in time if we mess up. Importantly, this means that mirroring your disk with something like RAID 1 is out. We need some other method that snapshots things. How often do we want to take snapshots? Maybe in our case with the photos we should have run a backup every week. If we lose 6 days and 23 hours of data, that’s fine and we can live with it. This is what’s called a Recovery Point Objective (RPO) in IT, and in real cases, it ranges from <30 seconds for critical financial institutions which really can’t afford to lose data, to 24 hours or more for some small enterprises (if they even have a disaster recovery strategy). Taking snapshots means that we have an increasing burden on our storage. With an RPO of 24 hours, you will end up having 7 snapshots per week. 30 per month. 365 per year, if you really don’t go and prune your snapshots. So you need to rotate your backups. Let’s say I go with the naive approach and decide to keep 14 days’ worth of snapshots. When I take a new snapshot, I delete the oldest one and I add the new one. Pretty simple, but this now forces me to have a watchful eye. Maybe I keep lots of data and can’t be bothered to check if something got corrupted in the past two weeks? But then again, I can’t just store a year’s worth of backups and they’re simply not relevant to me. What happened between day 2 and day 3 of the year has almost no significance when it’s day 364. So the granularity at which we take backups must change. The closer we are to today, the more frequent the snapshots. The further back, the less frequent the snapshots. So maybe we rotate our daily backups every 14 days, but also take weekly backups that we rotate every 7 weeks, and monthly backups we rotate every 12 months. This should be much more efficient. But again our complexity grows. We now have something called a GFS-rotated, snapshot-based backup. This list of adjectives will continue growing, as we’ll see in a bit. Maybe then you take a look at how MPEG compresses video, and get fascinated by how a calm scene in a movie, where the protagonist speaks but otherwise doesn’t move against a completely still background can be used for compressing video. You notice that videos are composed of frames that are mostly similar to each other, only changing with a certain movement
Backups aren't simple Software Engineer | FOSS Enthusiast | Homelabber --> Backups aren't simple Aleksandar Filipovski, 2026-09-16 See also: John Salvatier’s excellent blog, Reality has a surprising amount of detail I read a comment somewhere that stuck with me, that went something like this: “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” Trying to find the source for it for this blog, it turned out that every other sysadmin has his rehashed version of the quote, but the gist of it is the same everywhere. Data loss is something that happens more often than we’d hope, and most of us are woefully unprepared for when it hits us (which is almost always at the worst possible time). I can confirm that I had a similar experience once. When I was little we had pulled all our family photos from our home laptops and PCs onto an external hard drive, in order to free up some space. This worked beautifully until one day my dad wanted to use the drive as storage for our TV set-top box (one of these old things), and was prompted to format the drive. He went ahead with it, and the disk was reformatted. The index of files was deleted, and we were stuck with a nominally empty drive. It would be easy to blame him for screwing up, but it takes beginning a career in tech to realise that there is a series of errors that lead to this kind of mistake. Firstly, we had put all of our photos in one place and didn’t bother with backups. Secondly, most consumer-facing software usually has bold disclaimers telling you that formatting a disk means losing data (which the set-top box didn’t, terrible UI). Besides, why would you even expect a non-technical person to even have to know any of this? Thankfully we were able to get the photos restored, and it turned out to be a cheap lesson in handling data. You never keep important things in one place only. There’s about a million things that can go wrong. Your drive could die, it could be stolen, bits could rot in cold storage (hard drives have magnetic particles which can inexplicably shift, and SSDs are made of NAND transistors which leak electricity and over time, corrupt your data). So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good. This doesn’t cover the headaches of what a plugged in drive could do. Ransomware could encrypt your files, and you could do anything from an honest mistake like deleting the wrong file; up to catastrophic mistakes like running a script that overwrites everything with zeroes. So our backup should not be a mirror of the first drive, because we also want to be able to go back in time if we mess up. Importantly, this means that mirroring your disk with something like RAID 1 is out. We need some other method that snapshots things. How often do we want to take snapshots? Maybe in our case with the photos we should have run a backup every week. If we lose 6 days and 23 hours of data, that’s fine and we can live with it. This is what’s called a Recovery Point Objective (RPO) in IT, and in real cases, it ranges from <30 seconds for critical financial institutions which really can’t afford to lose data, to 24 hours or more for some small enterprises (if they even have a disaster recovery strategy). Taking snapshots means that we have an increasing burden on our storage. With an RPO of 24 hours, you will end up having 7 snapshots per week. 30 per month. 365 per year, if you really don’t go and prune your snapshots. So you need to rotate your backups. Let’s say I go with the naive approach and decide to keep 14 days’ worth of snapshots. When I take a new snapshot, I delete the oldest one and I add the new one. Pretty simple, but this now forces me to have a watchful eye. Maybe I keep lots of data and can’t be bothered to check if something got corrupted in the past two weeks? But then again, I can’t just store a year’s worth of backups and they’re simply not relevant to me. What happened between day 2 and day 3 of the year has almost no significance when it’s day 364. So the granularity at which we take backups must change. The closer we are to today, the more frequent the snapshots. The further back, the less frequent the snapshots. So maybe we rotate our daily backups every 14 days, but also take weekly backups that we rotate every 7 weeks, and monthly backups we rotate every 12 months. This should be much more efficient. But again our complexity grows. We now have something called a GFS-rotated, snapshot-based backup. This list of adjectives will continue growing, as we’ll see in a bit. Maybe then you take a look at how MPEG compresses video, and get fascinated by how a calm scene in a movie, where the protagonist speaks but otherwise doesn’t move against a completely still background can be used for compressing video. You notice that videos are composed of frames that are mostly similar to each other, only changing with a certain movement