My backup strategy

The first iteration

Backups are a pain in the ass. There are a billion ways to do them and even more ways in which your particular strategy can fail at any point. You might not even notice until you desperately need that one file back. Bitflips, bitrot and filesystems that need to study up on data integrity are very real things and might have awful consequences for the bits that make up the things you care about or can be exploited for shits ’n giggels.
I got introduced to this issue, like a lot of people I know by listening to the great John Siracusa (@siracusa) explain all the reasons why I should care about this by listening to Hypercritical episodes 56 and 57 in which he talks filesystems with Dan Benjamin (@danbenjamin).

John Siracusa talks about a bunch of filesystems, their strengths, weaknesses and why he has such strong feelings about HFS+. You should go and listen to these episodes if you haven’t yet, and if you have you should listen to them again! They are so good and have aged so well.
Ever since it was brought to my attention that data on any kind of persistent storage is still pretty fragile my entire family will gladly get up and leave the room whenever this topic comes up. I forced external storage onto my sister, mother father and everyone of my friends who were willing to listen to me, but whenever I sat down in front of one of their computers I was greeted by a similar message like this:

Last Time Machine backup 39 days ago

For obvious reason that wasn’t what I wanted to see. HFS+ is terrible at data integrity, I talked for days in order to get them an external storage device of some sort and in the end nobody was using them regularly. I had to look for an alternative, one that would take the entire backup flow, plugging in the storage device etc., and decisions about it out of reach of my family.

This has been entirely a very egoistic decision, but I rather fix this issue in this way than having to fight HFS+ later when all of my families data is shattered into pieces and I have to be the family tech support guy.

The decision wasn’t too easy though since I had to achieve all my goals with little to no budget and had to support macOS and different Windows versions. After a lot of googling around my decision fell on FreeNAS and unRAID. At first I didn’t entirely understand the difference between the two and decided on FreeNAS because of the absolutely brilliant build guides by Brian Moses. In hindsight I’m really happy I did so since Btrfs has some problems ZFS doesn’t have. I’m really happy with FreeNAS’ regular update cycle and all the tools available to me.

To be real for a second Btrfs is nothing to sneeze at, it is also designed around data integrity like ZFS, is stable, open source along side the Linux kernel on kernel.org! If I had no option but to run with it, it would have been fine but FreeNAS was chosen and so I built a low power backup server for my family running it. It is currently sitting under a desk in the office and is connected via CAT6 to everything else in the house. Every device in the house could backup to it via Wi-Fi or through ethernet and ever since I got the backup server setup I never have to see Time Machine being sad about how long the last update has been ever again.

The second iteration

Shortly after I moved in with my girlfriend and the FreeNAS server I had build for my parents was no longer available. I was in the same situation as a couple of months earlier: How am I going to backup my files in a safe and sane way. I wanted proper persistent storage for me and my girlfriend and once you’ve been spoiled with a Time Machine server just slowly backing up all your junk without you having to touch anything there is no way you will go back to external HDDs attached via USB or Thunderbolt.

I decided to build another FreeNAS server into an old desktop computer case from my dad since I had even less budget than before and put it somewhere in reach of my network and called it a day. That was around December first. Shortly before that I bought a new 5k iMac as my main workhorse thanks to my grandparents saving up all my life for educational purposes. Once getting my new shiny iMac I set everything up and transferred all my files over. My iTunes library with all the carefully placed ratings, changed names of titles I got off of a back of a truck and the handpicked artworks to go alongside it. I also transferred my photos, documents, programming projects and everything else. I then wiped my MacBook clean. You know the kind of

"sudo diskutil secureErase 0 /dev/disk0"

kind of erasing, so there was no way I could go back to all my stuff even I wanted to if something would happen to my shiny new iMac which had now become this single point of failure. If it failed and I had no backup, all my data would have been gone for good.

Do you know that people think they’re funny when they tweet this picture on December 31st?

On new years eve this actually happened to me. On the evening of December 31th I left my computer and didn’t turn it off or put it into standby. I celebrated new years eve with my girlfriend and once I came back to it on January 1st, HFS+ had corrupted the boot partition of OS X. It was corrupted to the point that it wouldn’t boot anymore and I couldn’t do anything about it. It would get past the POST and then fail with some crazy error message on screen. If I hadn’t had spent almost all of my last money on that stupid FreeNAS server all my data would have been gone now. No more music, no more photos, no more documents. All gone for good.

In late January I got myself a used Rittal 24U server rack off of [Ebay Kleinanzeigen][ebaykleinanzeigen] and really really cheap all purpose 4U rack cases for 64€. My parents still had scrap CAT6 sitting around from when they had it installed in their house and so I didn’t have to pay for the CAT6 I was going to run from my apartment down into the basement. In regards to the hardware that power the machine, I bought an Intel Skylake-gen Pentium G4400 clocked at 3.30Ghz, 8 GB of non-ECC DDR4 RAM and the cheapest motherboard that had 6 SATA III headers onboard. All opened boxes from Amazon warehouse sales. Due to the hard drives being the biggest item on the receipt I started out with used laptop hard drives I had laying around and later added new ones and transferred data over. I still use some of the origin ones and don’t mind that at all since ZFS is really good about data integrity and watching the underlying hardware. The second a drive fails I will receive an E-Mail alerting me about the degraded volume health.

My router+modem is setup is in my living room, from where I almost directly run the CAT6 cables down into the basement. After a short run of roughly 30 meter they are terminated in a 8 x 1 Gb patch pannel. My used and probably quite old Rittal 24U server rack isn’t a beauty but has worked out great for me. I only paid 150€ and I decided to buy and picked it up within two hours on a lazy Sunday 😊. The 19" chassis aren’t any prettier but they do their job and they were really cheap. They are 4U (quite big if you don’t know about this kind of stuff) in height and therefore quite versatile down the road. If I plan to upgrade at any point in time I can still use these to build other things without having to spend more money on 19" chassis.

cough Dual Xeon E5, dual GPU hackintosh that kicks any Mac Pros but in any regards cough cough

I know that this setup is quite overkill but I don’t do any backups into the cloud for any of my stuff. I probably should setup an offsite duplication of the ZFS volumes but I haven’t yet had the time to do such a thing and I would also need a lot of space for cheap. Backblaze has once again proven that they know what’s up when it comes to data storage with their new B2 Service. They go out of their way to make backups and data storage easier and more affordable than ever. I don’t personally use them but I looked at their services and can whole heartedly recommend them!

10.07.2016