Share your growing video storage strategy?

ManuCH · Dec 22, 2017

ManuCH said:
Btw, I suggest you look at an alternative S3 provider called Wasabi - wasabi.com. I'm not affiliated with them, but their S3 storage is much cheaper than Amazon's.

There are 2 catches:
1. you pay more for downloads than on Amazon, meaning that a data loss will cost you more, but regular usage won't
2. the minimum you pay for is 1TB for $4/month

Actually, I stand corrected. Wasabi S3's egress traffic (ie. when you download data from them) is cheaper than Amazon's ($0.04/GB vs $0.09/GB). I guess I'm even better off than I thought

So to sum it up, I always have at least 2 copies of my data. At minimum it's on my Mac and on a cloud storage (Wasabi in my case, either via Arq backup software or from the Synology NAS using HyperBackup), and at best it's on my Mac, on my NAS and in the cloud. Very important data (documents etc) are also on a sync service like Dropbox, but I don't use those for large videos.

I would never leave my data and my backups all in the same place. Keywords: house fire, robbery. If both the main data and the backup are gone, the backup was useless...

FASTFJR · Dec 22, 2017

ManuCH said:
Actually, I stand corrected. Wasabi S3's egress traffic (ie. when you download data from them) is cheaper than Amazon's ($0.04/GB vs $0.09/GB). I guess I'm even better off than I thought

So to sum it up, I always have at least 2 copies of my data. At minimum it's on my Mac and on a cloud storage (Wasabi in my case, either via Arq backup software or from the Synology NAS using HyperBackup), and at best it's on my Mac, on my NAS and in the cloud. Very important data (documents etc) are also on a sync service like Dropbox, but I don't use those for large videos.

I would never leave my data and my backups all in the same place. Keywords: house fire, robbery. If both the main data and the backup are gone, the backup was useless...

Synology NAS makes excellent machines. I'm a graphic artist for a large printing company. I have 5 of there NAS raid drives in our art department. They are very easy to set up and dam near bullet proof

Kilrah · Dec 22, 2017

I delete what's uninteresting, this includes trimming out the uninteresting parts of longer clips that have interesting stuff (without reencoding so as not to lose quality for the day I want to use them). That usually gets rid of 3/4 of a session's footage.

I tend to pile raw clips up a bit then go through them in a batch. 12TB storage on main machine.

marklyn · Dec 22, 2017

Kilrah said:
I delete what's uninteresting, this includes trimming out the uninteresting parts of longer clips that have interesting stuff (without reencoding so as not to lose quality for the day I want to use them). That usually gets rid of 3/4 of a session's footage.

I tend to pile raw clips up a bit then go through them in a batch. 12TB storage on main machine.

Kilrah, your strategy is more in line to my thinking. Many replies here have an almost commercial aspect to them, saving everything, redundancy, cloud backups, etc.. I do that for critical data but it wasn't my thought to do that for drone footage (unless I was using it for business). I guess I just don't see the point in saving all video unless someone here describes some value in it. That being said, I do offload all flight data, logs, & flight info files to a secure location. They are small and may have value in reviewing them for troubleshooting or diagnostic needs.

FatherXmas · Dec 22, 2017

I have 20TB here at the house and another 20TB at my son's house that any new or changed files get backed up to in the night.

Kilrah · Dec 22, 2017

marklyn said:
Many replies here have an almost commercial aspect to them, saving everything, redundancy, cloud backups, etc.. I do that for critical data but it wasn't my thought to do that for drone footage (unless I was using it for business).

Well I didn't mention it since to me it goes without saying, but there are 2 backups of those 12TB. I treat all my data the same way regardless of what it is since I noticed it took more time/effort/complexity to prioritize stuff differently.

Callum · Dec 22, 2017

Instead of videos of full flight I stop and start camera to get clips a few mins long.
After flights I put SD card into reader plugged directly into TV and watch through each clip and note down the file names of the ones I want to keep.

The good files get transferred to my Nas with raid 1 and scheduled nightly backup to Amazon S3.

I find smaller clips mean Im not holding long videos for the sake of a few seconds of good footage.

marklyn · Jun 22, 2018

During the past year I've come up with my own 'strategy' on how and what files I store for diagnostic usage or other reasons. It is a pain to do this after every flight and most people wouldn't go through this much trouble but for me it has helped me review flights, compare flights and changed flight missions to see my improvement.
I thought I'd share a snapshot if anyone wants to do something similar.

Jack Daw · Jun 22, 2018

One thing you might consider -- I do it -- is converting all your files to HEVC (aka H.265). You get the same quality for about half the file size. If there's a file that I really want to make absolutely sure is the best quality it can be, I leave it in h.264, but otherwise, HEVC is a great space-saver for things like test runs and experiments. There are a bunch of free converters available. I use one called Xmedia Recode.

Mossiback · Jun 22, 2018

The rule for data storage is; 3 different storage devices in 3 different locations. I feel that 3 is overkill since the odds of two devices corrupting your data at the same time is pretty low, but the important thing is to have a copy off-site. If using web storage make sure to use a very strong password.

I would also be very selective on what you record. Break recordings up into short segments; they are more interesting to watch and it reduces the risk of loosing it all if a file gets corrupted. Review it as soon as possible and then be more selective on what you save. Try to include a date and description in the name or folder. As you are finding out, it does not take long to amass many files and hours of recordings. Think how much you will have in five years. Are you going to sift through it? Is anyone?

wytnyt · Jun 22, 2018

Mossiback said:
The rule for data storage is; 3 different storage devices in 3 different locations. I feel that 3 is overkill since the odds of two devices corrupting your data at the same time is pretty low, but the important thing is to have a copy off-site. If using web storage make sure to use a very strong password.

I would also be very selective on what you record. Break recordings up into short segments; they are more interesting to watch and it reduces the risk of loosing it all if a file gets corrupted. Review it as soon as possible and then be more selective on what you save. Try to include a date and description in the name or folder. As you are finding out, it does not take long to amass many files and hours of recordings. Think how much you will have in five years. Are you going to sift through it? Is anyone?

i dont feel comfortable having my data on a remote or cloud site,after all they ARE ran by total strangers[think about it].
i keep my backups in a fireproof box thats stored in a fire resistant safe
im all good and no chance of prying eyes.

marklyn · Jun 22, 2018

I'm finding that I'm rendering everything I want with RaceRender at 1080 and uploading most of that to my YouTube library. About 20% of what I render stays on my 3 Tb drive and the rest of the videos are deleted since they have no learning value or scenic value or are bad footage in some way. When I render to 1080 I'm usually seeing a 3.5 gb file go down to about 1/3 of that size.
So between what I delete, render down and keep, render and upload to YT, my storage needs are really good.