Organizing and archiving your personal digital images
Apr 22, 2020 | By: Russ Ford
As a 20+ years IT professional there is one thing I know for certain about computers. They all eventually fail. Hard drives crash, floppy drives go bad, CDs and DVDs get corrupted. Even USB or solid state drives will eventually fail. How do you keep your data (your images in our case) safe? Well, the bad news is that the companies I worked for spent millions of dollars trying to protect their data and still had occasional failures. I doubt anyone reading this blog will have the resources to create multiple onsite and offsite backups over a T1 line to a disaster recovery site built into a mountain vault. The good news is that wilt today's technology there are good, inexpensive solutions to get reliable. safe, and easy setups for your data storage needs. The catch is that everyone has different needs. As a professional photographer I have millions of images I need to organize and store for 25 years or more. If you are just worried about your family vacation and special event photos it's much simpler. We are going to start with the simple solution first.
Simplify your data storage
First off before you all start sending me bigger, better solutions and telling me how wrong I am let me say that this is my opinion. There are too many ideas out there to mention them all but these are the ones that have worked for me. As technology changes and improves some of these methods will become outdated and need to be updated.
My opinion, NEVER use your desktop or laptop internal hard drive as the primary place to store your images. If you are running nightly backups or have an internal RAID array then okay, that might work. The problem is that the internal hard drives built into your computer are always spinning, they are always active, and they are getting read and written to all the time. Especially if you only have one drive as is the case in most laptops. The operating system is constantly reading and writing to that hard drive and as you can imagine, the more work something does, the faster it is going to wear out. As you delete files and create new one the system also leaves gaps or fragments the drive making it work even harder and slower. Ever noticed how a brand new out of the box computer is so fast but after months or years it gets slower and slower? Well that is a whole different post for another day. The gist of it is try to keep your important data files, like images, on a different hard drive than your system drive.
External Drives: One easy way to do that is to use external drives. When I first starting buying external drives they were about $500 for 256GB. Now you can get an 8TB external drive for under $150. I would suggest starting off with at least 2 of them. I put a small label on the first drive and call it RAW Files or Original Files and I put the date the first images were created. I store all of my images in folders with their creation date. For example everything shot today would go into a folder on my RAW Files drive called 200422. If those were the first files on the new hard drive I would write that date on the label. When the drive is almost full (I always try to leave at least 50GB free) write the last date on the label.
The second drive is going to be my Images or JPG drive. I put a label and also write the date of the first file folder. Once I have gone through and done my Lightroom editing on my RAW files or if you are only shooting JPG then I would narrow down the files to only the ones I want to keep. In Lightroom I then export the files I am keeping to JPG on my Images drive into the appropriate folder. This is where I would break the folders down into jobs or activities. I would keep the date as the beginning of the folder name but then add a brief description of the event (ie. 200422 John Smith Headshots, 200422 Jane Doe Family, 200422 Ford Family Visit to Antelope Island). Once the files have been exported or copied into their folders I can now work with the remaining files.
Note: I NEVER delete any images from my RAW drive. That might sound wasteful but come on, do you know how big 8TB is. I can afford the space. What I can't afford is to be cleaning up files and accidentally select a folder or folders and hit delete. If I never delete a file, I never accidentally delete something, simple as that.
Now that I have basically two copies of the files that are important to me I am feeling a little better. Still not good enough though. Ideally I would have a 3rd drive to backup the RAW drive again. Also I need to get a copy offsite of my most important images. An offsite backup system like Backblaze is great for backing up your system but not so much for backing up your images. If you accidentally or intentionally delete an image of folder of images and don't notice for a few days it may be hard to figure out when to restore from.
I much prefer having an offsite copy of my images in a usable online gallery. If you are an Amazon Prime user you have unlimited space to store your personal images on Amazon servers. You can access them by folder or image anytime you want. For professional or amateur photographers Smugmug or Zenfolio offer the ability to also store unlimited images online. You can even make them available to view and even allow others to download or purchase them if you want. There are free versions of some of these services but if you want access to the more advanced services there may be a small yearly or monthly fee.
All considered the cost of 2 or 3 external hard drives and an offsite backup solution is still under $500. The companies I worked for paid millions of dollars for much less. For me it's well worth the cost.
Note: I personally recommend Western Digital or Seagate Drives. For the price they have been very reliable for me. I won't say perfect however. I would never purchase the brand new biggest drives available until they have been well tested on the market for a while.
A better solution
Imagine a system where every byte of data, every file, every image gets written automatically on two distinct and separate drives. Even better a system that can backup 3 or 4 drives using only one drive's storage capacity giving you more space for less money. I have great news, these imaginary magical drives are real and affordable. A two drive bay goes for under $200. With two drives you could configure the drives into a RAID 1. Everything you store on disk 1 also gets mirrored onto disk 2. If one drive failes you get a warning on the device, pull the bad drive out and replace it with a same size drive and the system automatically re-mirrors the drives, usually in only a few hours with no effort on your part. The advantage of this system is for a very inexpensive price you can get a fault tolerant disk storage system that is pretty much worry free. You can buy external drives that are already setup as mirrored or buy an enclosure and put two internal drives of whatever size you choose up to the limits of the enclosure. Performance is also enhanced in RAID 1 since the computer can read data from either drive. The disadvantage is you are spending more in the long run because that mirroring is 1 to 1. You are only getting 1 drives worth of storage out of two drives.
Another option is a RAID 5. There are other RAID options out there but the most common and easiest to setup and use are the RAID 1 and RAID 5. A RAID 5 array uses a minimum of 3 disks but can have more in the enclosure. RAID 5 has the advantage of RAID 1 in that all data is mirrored using a very complex algorithm that I don't really understand. Let's just say that it can magically store more than it should be able to and mirror the data to multiple drives. If one drive fails all the data in the array is still there and safe. Pull the bad drive out, replace it and magic happens and automatically re-mirrors all the data back to the new drive. Pretty cool. Enclosure for 4 drives is about $500 plus internal drives about $150 each for 8TB so $1100 for 24TB of storage. That enough for about 50 million images give or take 10 million or so depending on your camera. Smaller drives would of course cost less but anything less than 2TB drives would be a waste I think.
I would still buy a few external drives to archive your images off the RAID as they age out and give you an "offline" backup of original files (something that is not plugged in all the time). An "offsite" backup is also still a great idea (through Amazon Prime or Smugmug) in case of fire, flood, earthquake, or other natural disaster in your home or business.
Start small and expand as needed
If you are just getting started in photography and you don't have a lot of images to worry about you are lucky, you can do it right from the beginning. If you already have bucket loads or virtual shoe boxes of digital files that you don't know what to do with then start small. Pickup a couple of external drives and start organizing a few at a time onto those new drives. If the old ones are already on smaller external drives don't delete them, just copy the files and when you have them all organized put the older drives in a box on a shelf for just in case. Don't delete the files from your desktop or Laptop until you are absolutely sure that there copies successfully. Use your file manager on Windows or Mac and make sure that the folders have the same number of bytes of data (in Windows right click on the folder and look at Properties). Actually check some of the images to make sure that you can see them as images. Don't rename them or change them in any way as you copy them.
Whatever you do start doing it now. Remember, every hard drive you have will eventually fail. I really hate hearing stories about people who have lost all their family digital files but unfortunately I hear those stores all the time. Email me if you have any questions about getting started firstname.lastname@example.org. Good luck and have fun.