aows

View Original

My hard drive broke with 4TB worth of data in it: lessons learned from the disaster

They say you don't know how good your backup system is until something bad happens. Well, that something bad happened to me a few days ago, and I regret some of the decisions I made about my backup strategy.

Don't get me wrong: I didn't lose any data. But the process to recover it is taking much more time and effort than I thought.

What broke and what I (almost) lost

It wasn't my small hard drive that broke, of course. It was the big one, with 3.5TB worth of images and videos in it.

The images were never a problem, since I use Adobe Lightroom CC. All of them (including the originals) were backed up in the Adobe cloud and I was still able to work on them as they were being downloaded to my new hard drive.

The problem was the rest of the data: videos. I keep every single clip I record, probably a mistake.

The 4k videos I took during the American Road trip alone take more than 2 of those 3.5TB. I don't use 4k since then (for that and other reasons that I'll talk about soon), but even 1080p videos can take a lot of space.

The cloud to the rescue! Kind of

The only copy I had of all those videos was (and still is) in Backblaze. They offer a great service for the cost, and it saved me this time. But recovering the data is a real pain.

You can either order a new hard drive from them with all your data preloaded in it, or download it yourself.

I'm not in the US, and I don't want to pay almost twice the market price for a hard drive, so I decided to download the data.

You'd think you could click a button and start downloading all of it. You'd be wrong, though. You have to select which folders you want to recover, manually, and then Backblaze will create a zip file that you can download.

Downloading the whole thing is not an option.

I've spent hours creating and downloading those files these past few days -and I still have 1.2TB of data left.

Moving to redundant local copies

Backing up to the cloud just doesn't cut it for large amounts of data. Even less if it's your only copy.

When disaster happens, all of a sudden your data isn't available and won't be for a while -hopefully you don't need it right away.

Also, the sheer time that will take you to download all of it means you have to keep your computer on 24/7 for days.

Don't forget about all the time your computer (and hard drives! This was probably the reason why my drive failed so soon) had to be on to upload that same data.

I'm still using Adobe Cloud to store an extra copy of my images. It worked great and I had all the originals (almost 400GB) locally in just a few hours.

For videos, though, I'm moving to duplicated hard drives.

I purchased three new hard drives:

  • one 4TB hard drive to replace the broken one,
  • another 4TB hard drive as a backup for the first one, and
  • a 512GB SSD where I'll keep the images and videos I'm working on to avoid making the spinning disks work too much

Choose your own poison

There's no perfect backup system, at least for an affordable price.

Last year, I decided to go the "one hard drive + cloud" route because I was going to be on the move.

Pros: traveling makes losing stuff more likely to happen, and I was afraid of misplacing my backpack or having it stolen (and all the data with it). The cloud can't be lost.

Cons: in the end, I didn't have reliable Internet connections during most the trip, thus I spent weeks with just one copy of thousands of videos from the trip of a lifetime. Not a good idea.

Now, I'm opting for the "local redundancy" route.

Pros: I don't need an Internet connection to have redundancy, and I'll still be able to access my data right away if something ever happens...

Cons: ...unless that "something" is losing the hard drives or having them stolen. Hard to avoid because I usually have the hard drives in the same bag while traveling.

The (maybe) best solution for traveling photographers

Right now, I'm on a trip in Portugal.

To avoid a complete loss in case I misplace my luggage or a thief targets me, I left one of my main two hard drives at AOWS HQ.

I don't want to have just one copy of my images and videos, though, so I'll travel with yet another hard drive in my bag. A 2TB unit I already owned but was barely using.

Every night, I copy every video and photo to those two hard drives.

This, along with an additional copy in the cloud, would be an almost perfect solution for a traveler photographer.

I will no longer be using Backblaze, though: I don't have the patience to go through all this again and I don't want to keep my computer and / or hard drives on for entire days anymore either.

Besides, the house where we are at has (not surprisingly!) a very slow connection.

TL;DR

My big hard drive broke and I almost lost 3.5TB of images and videos.

They are safe, but they are far away, in the cloud. It's taking me forever to get them back.

From now on, I'll be using multiple hard drives to keep my stuff backed up.