Development time is money, therefore I RAID
In the line of work I am in, development time is money! This week marks the third year after I retired my Apple MacBook Pro as my primary system for a workstation that I built from parts from MicroCenter, a retailer that focuses on serving the gaming community. I also use a RAID 1 mirrored disk as well as backups for my data.
You may ask how it is that in 2020 I am doing my work on a PC tower when those went out of style for all but supercomputing and gaming around 1999? I have owned my share of laptops over the years since high school and going into Computer Science at Georgia Tech. I still own multiple laptops, but in 2017 I decided that the time to restore after a hardware failure was just too great.
In security, we are concerned with confidentiality, integrity, and availability. When a developer’s laptop goes down, it is unavailable for the work that was intended for the day.
To put this in perspective, for the better part of 10 years I worked 16 hours a day on code. I ran my laptops hard, hard enough that they predictably had hardware failures while still under either AppleCare or the American Express extended warranty. It’s nice to know that the hardware repair is covered financially, but when the soldered on SSD fails or the system board has an issue the turn around time is at least three (3) days. This means as soon as there is a problem, you have to buy a new computer anyway and start the process of restoring from your backups. This is what the screen on my 2014 MacBook Pro looked like that fateful April day:
This was not the first time I woke up to a computer that would not boot. In fact it was only 8 months earlier that I had purchased this system to replace my previous MacBook Pro that was having thermal issues out of warranty. In both cases, this meant cancelling onsite meetings for the day and losing at least $800 in billable hours each time - ouch.
Since this was my only Mac OS system at the time, and the repair was to take several days, I bought a Mac Mini on the spot and then had to restore from my local time capsule backup, a process that took many, many hours. Thankfully that backup saved me and I was able to resume work the next day. If I had had to restore from the online offsite backup, things would have taken far longer. After the system was repaired, I used it for another two and a half years until the day before Christmas 2017, the MacBook Pro started kernel panicking while doing very basic tasks. Prepared, I had a full system local backup and took it to the last genius bar appointment on Christmas Eve. It was out of AppleCare, but I was given a quote to repair and that was covered by the extended warranty on my American Express credit card. I let Apple fix the computer, which took several days. That night I scrambled to get an old laptop running Linux so that I would be able to support my customers over Christmas if a production issue happened. The day after Christmas, I decided enough was enough and went to MicroCenter in Duluth (Metro Atlanta area) and bought the parts to build my own PC. The idea is that if something breaks I have the ability to source parts locally and repair unlike with a soldered on laptop.
I still have two laptops, a Dell XPS 13 Developer Edition running Linux, and the aforementioned MacBook Pro to tinker around with Mac OS and Safari. I decided that I didn’t need a laptop to be my only computer ever again. I do my work on the workstation that I built that day. It has a SSD primary drive and two 4 TB HDD in a RAID 1 mirror. I keep the home directory on the RAID. This means if the system fails, I have multiple avenues to get back up and running within hours rather than days.
I have used rsync and/or Unison (available for Mac OS, Unix, and Windows) to keep project files synced between the Linux laptop and my workstation. This lets me grab the laptop and go and am still able to work away from my home office. It is important to remember that RAID and mirrored copies protect against data loss due to drive failure, and shortens recovery time to get back to work, but do not protect against deleting files you didn’t mean to. That’s what backups and Git (for source code) provide.
At the moment my configuration is as follows:
- Workstation PC running Ubuntu Linux
- Primary OS disk is a 500 GB SATA SSD
- Home Directory is hosted on redundant 4 TB HHD in a RAID 1 mirror
- Ubuntu Linux Laptop syncs a subset of directories from the Linux Workstation via the Unison utility. The storage available on the laptop is smaller than my home directory on the main system.
- Mac OS laptop uses external USB drives and mounts data on FreeNAS server
- Backups to a FreeNAS ZFS server on the local office network
- Further encrypted offsite backups to CrashPlan Pro to protect against the chance that a catastrophic failure takes out all my systems such as a lightning strike or fire
As you can see, I have invested heavily in redundancy and speed-to-recovery for my computing infrastructure. I make my living building software and providing support. I’ve built a business reputation over decades that could easily be lost when I can’t work because of computer problems. Being able to recover from the predictable but irregular computer hardware failure nearly instantly is a must.
This is a defense in depth strategy against loss of the availability security pillar. Cloud storage and online backups are a must, but restoring from them can take a considerable amount of time. That’s why I have invested in local redundancy via RAID workstation and RAID backup server. The sync with the laptop also serves to provide even faster recovery time in the case of a system failure. This way I’m not sitting for hours waiting on a new computer to restore from backup before I can keep working. Time is money after all.
I also protect the confidentiality of my data through full disk encryption on all my drives, especially SSD, but that is a topic for another blog post.