Cloud Storage Solutions

My belabored attempt to break down the practicality of using cloud based backup solutions.  There are probably services out there that I have not mentioned but this is just a few that come to mind.


Edited 3/7/2011 to reflect Google Storage for Developers lower prices.


So I had a drive fail this morning on my office workstation, but luckily I had everything mirrored on a NAS (network attached storage) device at home.  The drive is still under warranty and the data was backed up so life will go on.  But now I have to go to the trouble of:

• Calling the manufacturer to get a new drive sent out to me.  I opted to pay $10 to have them ship me a new drive and a pre-paid envelope to return the old one in so that I would only have to take my workstation out of service once.
• Wait for new drive to arrive and hope that nothing happens to the other drive while I wait.
• Install new drive in the case, mount format and test the drive.
• Copy data over to new drive.
• Mail back the old drive.

This is a lot to do and I have a drive fail in one of our machines roughly once a year (granted we have a lot of machines) so this is not something I am new with.  Still it is a real pain so I looked into cloud based storage solutions.  The current setup is really quite roomy and we store a lot of music and videos.  It breaks down something like this:

• 180GB of music (30,000 songs)
• 150GB of movies (125 movies)
• 70GB of pictures and home movies (7GB jpg, 10GB raw, 52GB mov, and 3GB other)
• 100GB of backup for laptops
• 100GB of backup for work

Add a little padding and you are looking at roughly 1TB of space (OK so that is a lot of padding but I am planning for growth).  We have already started taking advantage of some free and paid solutions for backing things up over the net.

Flickr pro account -- we have been using this for a long time and it has served us well for uploading jpeg files although I was always a little remiss that we couldn't upload RAW files.  It really does not matter to me as much now as I don't have time to mess with RAW processing and just shoot jpeg anyways.  This gives us unlimited storage for under $30 a year.  The uploader works pretty well and I like the simplicity of the site.

Picassa -- I really only started using this more when I started using Google+.  The nice thing is that photos are uploaded automatically when I take them on my phone, and now that my phone takes pictures that are almost as good as the ones we take with out point and shoot camera I find that I am uploading to Flickr less often.  I have thought about upgrading our account here to have more storage but the prices are still a little steep as we would have to pay $50 for 200GB of storage and it is likely that we will need to upgrade to the 400GB plan in the future costing us $100 per year.  

Edit: after combing through the files and filtering out the movies from the jpg and raw files we would only really need to keep the jpg versions backed up and the movies would be on YouTube so that means that our storage requirements drop down to what we could fit in the 20GB for $5 per year plan.  We could even go to the 80GB for $20 per year and have plenty of room to grow.  The only think I am not sure about is if Picassa will store the original file size or if it is reduced.

YouTube -- For family videos you really can't beat it.  Unlimited uploads and the interface/quality is great.

Dropbox -- I have about 6GB of storage space on the free offering, and used to use it quite a bit for projects that I worked on.  I think this is My wife's favorite because it is dead simple to use.  We could upgrade to the paid versions but for $100/$200/$760 a year it is a little pricey.

Google Docs -- this service now allows uploading of arbitrary files, and I use it quite a bit for word processing and presentations but because there is no native client for syncing and uploading files I have not made use of it yet as a storage device.

Google Music -- we are relatively new to this but have enjoyed it quite a bit lately.  It allows us to sync the music to our phones, listen in the browser and if we need to we can actually download all the music to our local device.  The real drawback here is that you are limited to 20,000 songs which is a lot unless you consider that we have 10,000 more than that.

Add to that my use of Google Code, Github, and Bitbucket (a great service I might add!) and we really are a pretty cloud friendly family.  But what would it cost to get rid of the need for our NAS and extra backup drives?  We need a solution that would allow all of our music, videos and backups online.

Price per GB annual costs

Google -- is about $0.25 per GB for any file type in blocks from 20-400GB.
Amazon Cloud Drive -- is $1 per GB, BUT they allow free storage of mp3 files which saves us from having to pay for around 200GB of storage.

Dropbox is also around $1 per GB until you get up to the teams level (but that cost is prohibitive for most).

Amazon S3 Storage -- (the backend for the Amazon Cloud Drive and Dropbox) is $1.50.  Obviously we would need to store considerably more to get the competitive pricing that Amazon gives Dropbox and its self.
Google Cloud Storage -- (the backend for the apps storage etc) is also $1.50 $1.44 per GB per year.  It seems as if the storage folks have all done their math and no one can really go much lower but Google is not about to be outdone.  Of course if you have 500TB of data and a budget of $535,941.12 per year to spend on storage then you can store your data for the low low rate of $1.04 per GB per year.

My own drives in A QNAP box -- By using RAID 1 (TS-419P+ Turbo NAS), 2TB drives, and figuring an absurd failure rate of one drive per year...  $0.07 per GB.  Of course the time setting it up and the enclosure hardware should count in this but I am going to assume that those costs will balance out with needing to configure all kinds of remote services and the time it would take to download a recovery.

So where does this leave me?

Cloud storage is not cheap and it shouldn't be.  For everything I can do at home to keep my data safe (including keeping a copy at my office) it does not compare to the redundancy and protection that it would receive in a data center.  That being said, for the applications that I have in mind I am still better off with the cheaper DIY solution.

What is the cheapest way that I could slice and dice this?  Well I could split my music up across 3 Google Music accounts, use YouTube for my home movies, Picasa for my photos ($20 should do it) and once all the media files are removed from the laptop backup we could take care of that with Dropbox or what's left for storage in Google Docs.  That just leaves the elephant in the room, the movies.  These files range from 1 to 2.5GB each and are above the file size limits for Google Docs.  Which means that I can do one of two things.  Split them into parts (not hard but time consuming) and store them in Google Docs for $50 per year OR pony up for one of the $1 per GB storage solutions.

I can buy plenty of hard drives for $70 to $200 per year.

What would I like to see?

I would love to see locker services like Google Music flourish.  Something that I have not been completely impressed with is taking advantage of what can be done with de-duplication and reducing the costs of uploads.  To test this I uploaded the same song to my account and my wife's account (same file from the same system even) and I had to wait for it to be uploaded both times.  Better duplication detection would reduce storage and ingress costs for all of these services that could be passed on to the users.  This would also make moving my music collection to the cloud more feasible.

Movies are another thing that I would love to see addressed by the file-locker type service.  Frankly the selection of streaming services suck (netflix and amazon I am looking at you!) but how great would it be if I could use a netflix media locker and enjoy the same streaming platform to stream my own movies back to me!?!

Once my movies and music is taken care of I would be all set.  My hunch is that most of the advancements I would like to see are more of policy challenges (disrupting old business models) rather than technical ones.

A side note on high capacity drives

Why have I replaced 350GB, 500GB, and 1.5TB drives and I have a much older 160GB drive that sees much more sustained use and it is still plugging along?