Data Overload!

[Chaosrider]

I have never in my life been engaged in an activity that accumulates so much data, so fast. And I'm still at 1080p!

I'm busily deleting a bunch of stuff that I already have backed up 5 times, but that's just a stopgap measure. I've got plenty of external storage that I can shuffle around and optimize, but I really need to develop a data strategy.

And the video editing just takes for ever. I find myself literally having to wait for the machine to catch up.

Fortunately (sort of...) I need to get a new computer anyway, so I can design the new system around that.

My "Drones" folder is already up to 300GB.

I'm a PC guy, not a Mac guy. What kind of configuration would ya'll recommend? The video editing doesn't have to scream, but I don't want it to crawl like it does now. I'm assuming that I'll need a RAID array for a proper real-time backup.

One perhaps unusual constraint is that I have a really rotten internet connection, so the system has to self-contained. It can't rely on the internet for normal operations.

Suggestions?

Thx!

TCS

 

[MarkusAtHome]

I'm assuming that I'll need a RAID array for a proper real-time backup.

Be aware, that RAID is not a substitute for a backup. A RAID is only a safeguard for a damaged HDD/SSD.

I have a NAS (RAID 5) with all my personal files. I pull a differential backup of my NAS every now and then (depending on the amount of changes, sometimes every other day) on to an external HDD. This backup HDD I keep at another location (not at home where the NAS is). In case I “loose” the NAS I still have all data available.

 

[tlswift58]

If building a PC - the AMD's chips are comparable to Intel's and cheaper. If you can go to a Ryzen 7 series, they are like 8 core and are fast (

AMD Ryzen 7 5800X - Ryzen 7 5000 Series Vermeer (Zen 3) 8-Core 3.8 GHz Socket AM4 105W Desktop - $400 at Newegg​

Get a good motherboard, not a cheapo if you keep it for years like I do. Get a minimum of 16GB's RAM, but 32 is better. A good video card is going to be about $200-$500.

As for storage, a NAS may be something to look at, but I run a dock (USB extremal) with 2 harddrives. Currently both are 2TB's and I have maybe 650GB's on them. Larger drives are available and go with them - 4-8TB's if you do a lot of video at higher than 1080p. I have one drive as my main drive for storage and the other is a duplicate. And yes, I have lost a drive in the past and having the 2md one saved me should a software recovery not be possible. But, with drives so cheap - I simply pulled the bad one out, put in the new one, and copied files over from good drive.

Backup is even more critical if you have an SSD as a main drive.

With the "chip" shortage" all components have gone sky high. So building one will cost more until that issue subsides some. I like building my PC's as I build what I want and don't have to sacrifice something to buy a "built" one.

I'd also think about a water cooling system. They do work well in keeping a CPU cooler and running better / faster. Lots of fans as well all over the case too.

 

[Chaosrider]

Be aware, that RAID is not a substitute for a backup. A RAID is only a safeguard for a damaged HDD/SSD.

I have a NAS (RAID 5) with all my personal files. I pull a differential backup of my NAS every now and then (depending on the amount of changes, sometimes every other day) on to an external HDD. This backup HDD I keep at another location (not at home where the NAS is). In case I “loose” the NAS I still have all data available.

RAID is certainly a form of backup, although I'd continue doing the regular back-ups that I already do.

I do my normal back-ups to external HDDs, and rotate them.

Thx,

TCS

 

[dave canada]

If building a PC - the AMD's chips are comparable to Intel's and cheaper. If you can go to a Ryzen 7 series, they are like 8 core and are fast (

AMD Ryzen 7 5800X - Ryzen 7 5000 Series Vermeer (Zen 3) 8-Core 3.8 GHz Socket AM4 105W Desktop - $400 at Newegg​

Get a good motherboard, not a cheapo if you keep it for years like I do. Get a minimum of 16GB's RAM, but 32 is better. A good video card is going to be about $200-$500.

As for storage, a NAS may be something to look at, but I run a dock (USB extremal) with 2 harddrives. Currently both are 2TB's and I have maybe 650GB's on them. Larger drives are available and go with them - 4-8TB's if you do a lot of video at higher than 1080p. I have one drive as my main drive for storage and the other is a duplicate. And yes, I have lost a drive in the past and having the 2md one saved me should a software recovery not be possible. But, with drives so cheap - I simply pulled the bad one out, put in the new one, and copied files over from good drive.

Backup is even more critical if you have an SSD as a main drive.

With the "chip" shortage" all components have gone sky high. So building one will cost more until that issue subsides some. I like building my PC's as I build what I want and don't have to sacrifice something to buy a "built" one.

I'd also think about a water cooling system. They do work well in keeping a CPU cooler and running better / faster. Lots of fans as well all over the case too.

sir could i I consult you when I can or need a new computer

 

[eEridani]

I only use parity raid on a server class OS with ECC memory for my backup needs. The parity checking along the data path for backup file IO is a fundamental part of ensuring the backup is viable when you need it. Most NAS boxes don't have secure data paths, so the fact some can do parity raid is sort of like jumping into a fire net that is laying on the ground.

A raid on the machine you are working on is good to get data rates up and can help with file integrity if its parity raid, but isn't what most would call a backup method.

Backing up to a non-raid single drive is problematic. With uncorrectable data error rates of 1E-14 being common, if you do the math you'll see an error is likely if you simply write then read a full 10 TB of data. So some form of parity to recover from a read or write error becomes important: parity raid is better than no raid.

 

[Prop Wash]

To save some space on less used files, have you considered a "ZIP" program to compress your data? Just a thought.

 

[CanadaDrone]

I have never in my life been engaged in an activity that accumulates so much data, so fast. And I'm still at 1080p!

I'm busily deleting a bunch of stuff that I already have backed up 5 times, but that's just a stopgap measure. I've got plenty of external storage that I can shuffle around and optimize, but I really need to develop a data strategy.

And the video editing just takes for ever. I find myself literally having to wait for the machine to catch up.

Fortunately (sort of...) I need to get a new computer anyway, so I can design the new system around that.

My "Drones" folder is already up to 300GB.

I'm a PC guy, not a Mac guy. What kind of configuration would ya'll recommend? The video editing doesn't have to scream, but I don't want it to crawl like it does now. I'm assuming that I'll need a RAID array for a proper real-time backup.

One perhaps unusual constraint is that I have a really rotten internet connection, so the system has to self-contained. It can't rely on the internet for normal operations.

Suggestions?

Thx!

TCS

These days, there is no real need for more than a single local backup (2X your data) - if your house burns down it doesn't matter if you have 100 local backups. I personally use Blackblaze, which is around $4/mo for unlimited storage and easy recovery options. It's also automatic. 300GB is hardly anything though so I am wondering if I am misunderstanding you.

Consolidate your local hard drives to fewer larger ones. Look for sales on external drives 10TB or above - most of the time they have very high end drives inside for about half the price of what they cost as standalone drives. For example all the Seagate 10TB external drives have very high end Baraccuda Pros or Exos drives in them - I've bought several for $199 CAD and those drives bought separately are close to $500 CAD. It takes 30 seconds to pop them out of their enclosure and then you can install them in your PC to be used as normal.

If your video editing takes forever, what kind of computer are you using? If you want to share a budget I would be happy to put together a build for you. Your problem is going to be finding a GPU in stock though, and many video editing tasks are handed over to the GPU as it can do certain things much faster than a CPU. In the meantime, if you use a program like Davinci Resolve, you can edit in proxy mode at a lower resolution than your output resolution to help speed things up, but if you're already struggling with 1080P I suspect your PC is simply overdue for an update

 

[eEridani]

Keep in mind a typical spinning HDD will move data at 100MB/s if and only if the file is large, not fragmented, and has only one process using the drive. The moment anything else uses the drive, seeks consume available time, data transfer is shared between processes, and data rates plummet.

Spinning drives shouldn't be used for editing.

 

[Flycaster]

You can try to build up a PC, but getting the GPU card, may be a bit dicey. Chip shortage/issues...
What may be easier, is to look at the "prebuilt" PC's out there, WITH a separate and dedicated video card with at least 6 gb RAM. Don't think that an "integtrated video card" that is on the main motherboard will cut it.
I picked up an Asus gaming machine laptop from Best Buy a couple of years ago, and works fine.
8GB dedicated Nvidea video card, 12 gb RAM and 2 m.2 SSD drives....about 1K$...

 

[Chaosrider]

These days, there is no real need for more than a single local backup (2X your data) - if your house burns down it doesn't matter if you have 100 local backups. I personally use Blackblaze, which is around $4/mo for unlimited storage and easy recovery options. It's also automatic. 300GB is hardly anything though so I am wondering if I am misunderstanding you.

Consolidate your local hard drives to fewer larger ones. Look for sales on external drives 10TB or above - most of the time they have very high end drives inside for about half the price of what they cost as standalone drives. For example all the Seagate 10TB external drives have very high end Baraccuda Pros or Exos drives in them - I've bought several for $199 CAD and those drives bought separately are close to $500 CAD. It takes 30 seconds to pop them out of their enclosure and then you can install them in your PC to be used as normal.

If your video editing takes forever, what kind of computer are you using? If you want to share a budget I would be happy to put together a build for you. Your problem is going to be finding a GPU in stock though, and many video editing tasks are handed over to the GPU as it can do certain things much faster than a CPU. In the meantime, if you use a program like Davinci Resolve, you can edit in proxy mode at a lower resolution than your output resolution to help speed things up, but if you're already struggling with 1080P I suspect your PC is simply overdue for an update

My PC is very seriously in need of an update! I'm not going to build it myself, so I won't have to personally find a GPU, but I'm sure it's still going to byte me in the price. I'm pretty much ready to buy it, once I get the configuration settled.

On-line back-ups are problematic for me because my internet service...particularly my upload speeds...is seriously bad. But putting one in the laundry room...which is built into the cliffside and less likely to burn...provides some margin over the main back-up that I just keep next to the machine.

I'm only at 300 GB because I just got started down the drone path...

;-)

I'll check out the larger external HDDs; sounds like they've gotten cheaper since the last time I looked.

I'm quite happy with Wondershare. The problem is my computer.

I'm just a consumer video editor (and brand new at that), so I don't need a Blazing Saddles machine. If any of you have a reference configuration for a new PC, a spec sheet, I'd very seriously appreciate it! Since drone video editing is by far my largest demand use, if it works for that, everything else will work easily.

Thx!

TCS

 

[CanadaDrone]

My PC is very seriously in need of an update! I'm not going to build it myself, so I won't have to personally find a GPU, but I'm sure it's still going to byte me in the price. I'm pretty much ready to buy it, once I get the configuration settled.

On-line back-ups are problematic for me because my internet service...particularly my upload speeds...is seriously bad. But putting one in the laundry room...which is built into the cliffside and less likely to burn...provides some margin over the main back-up that I just keep next to the machine.

I'm only at 300 GB because I just got started down the drone path...

;-)

I'll check out the larger external HDDs; sounds like they've gotten cheaper since the last time I looked.

I'm quite happy with Wondershare. The problem is my computer.

I'm just a consumer video editor (and brand new at that), so I don't need a Blazing Saddles machine. If any of you have a reference configuration for a new PC, a spec sheet, I'd very seriously appreciate it! Since drone video editing is by far my largest demand use, if it works for that, everything else will work easily.

Thx!

TCS

I would generally advise against prebuilts unless a reputable builder has made them. The reason for this is they usually use garbage parts for motherboards and power supplies, second-rate GPUs (just because a prebuilt says it has, say, a RTX 3060 does not mean it's the same as one made buy a quality third party vendor such as ASUS, EVGA, etc.), and proprietary parts that are not easily replaced if they fail. On top of this they usually just aren't built very well in terms of cable management, thermal compound application, CPU cooler installation, etc. DELL is an example of this, they have been caught putting out some pretty horrible prebuilt desktops (their laptops are much better). I don't know what's available in your area, but usually there are services that will build you a PC to spec so you get the best of both worlds with quality parts and expert assembly.

As a "mid-range" video editing PC, this is roughly what I would recommend if purchasing today:

https://pcpartpicker.com/list/F8yw9N

Obviously things can be tweaked one way or another to suit your personal preferences/budget. I just chose a random RTX 3060 GPU because you will probably end up having to just buy whatever you can find in stock for a GPU closest to your budget. You can probably use your existing hard drives for raw storage, or just find a good deal on an external drive and shuck it. I'm assuming you are willing use your existing peripherals as well.

The thing with building a PC is quite often you only have to spend a tiny bit more money to get a huge increase in performance, so I did the best I could to pick at the point of diminishing return with video editing in mind.

 

[Flycaster]

Always go with a name brand,
Asus ROG, or Alienware (owned by Dell, but does ok on the gaming laptop platforms, their desktops went to the "pile" a number of years ago and never quite came back).
If the machine you are looking at can handle the game Crysis (the original, not the remastered/downsampled), then you would good to go...
It's the only time my Asus ROG fans do come on....

 

[VenomXts]

I am with you, couple of things I have learned..

1. Only shoot video if you want to.... its the space killer.
2. Big slow drives are cheap, im used to taking off 40 gigs at a time but still takes lots of time... Edit on a faster drive don't store.
3. for your finished stuff you put time into, cloud storage... (one drive, drop box etc) I don't back up everything but if I processed something its backed up.
4. Not sure of your computer specs but one thing I learned the hard way using davinci resolve is cpu encoding on windows is slow! I got the version that used my gpu (1080 ti) and my god that saves some times.
5. last but not least 4k is cool... but only if you punch into the video...

Just some unasked for advice. I am currently looking at bigger drives and a new computer for the next 5 years.

 

[Chaosrider]

I would generally advise against prebuilts unless a reputable builder has made them. The reason for this is they usually use garbage parts for motherboards and power supplies, second-rate GPUs (just because a prebuilt says it has, say, a RTX 3060 does not mean it's the same as one made buy a quality third party vendor such as ASUS, EVGA, etc.), and proprietary parts that are not easily replaced if they fail. On top of this they usually just aren't built very well in terms of cable management, thermal compound application, CPU cooler installation, etc. DELL is an example of this, they have been caught putting out some pretty horrible prebuilt desktops (their laptops are much better). I don't know what's available in your area, but usually there are services that will build you a PC to spec so you get the best of both worlds with quality parts and expert assembly.

As a "mid-range" video editing PC, this is roughly what I would recommend if purchasing today:

https://pcpartpicker.com/list/F8yw9N

Obviously things can be tweaked one way or another to suit your personal preferences/budget. I just chose a random RTX 3060 GPU because you will probably end up having to just buy whatever you can find in stock for a GPU closest to your budget. You can probably use your existing hard drives for raw storage, or just find a good deal on an external drive and shuck it. I'm assuming you are willing use your existing peripherals as well.

The thing with building a PC is quite often you only have to spend a tiny bit more money to get a huge increase in performance, so I did the best I could to pick at the point of diminishing return with video editing in mind.

Hmmm...$1350 not including the video card, and just for the pile of parts, unbuilt. Pretty pricey!

I'm not going to be building my own machine, but this is certainly a good framework for what I need to get.

I'm also not going to get a random pre-built from someone I don't know. I'm thinking Dell, Best Buy, somebody like that. There's a local guy with a storefront who's been around for quite a while, so that might be an option.

I'm certainly not flush with money, but at this point, I have more money than I have time for this project.

Many thanks!



TCS

 

[CanadaDrone]

Hmmm...$1350 not including the video card, and just for the pile of parts, unbuilt. Pretty pricey!

I'm not going to be building my own machine, but this is certainly a good framework for what I need to get.

I'm also not going to get a random pre-built from someone I don't know. I'm thinking Dell, Best Buy, somebody like that. There's a local guy with a storefront who's been around for quite a while, so that might be an option.

I'm certainly not flush with money, but at this point, I have more money than I have time for this project.

Many thanks!



TCS

Well you didn't share a budget So I picked a very mid-range system. It's a reference point if nothing else. A proper video editing workstation would be far more expensive than that.

Also as I mentioned, you don't need to build it yourself even if you are picking your own parts - there are very likely services in your area that will build you a custom PC, and it will be so much better than anything you find as a pre-built, especially from a big box store. There are online services that will do this for you as well.

Dell/Best Buy is perhaps the worst place to buy a desktop PC, for the reasons mentioned in my previous post. If you have a local guy who does this sort of thing, almost certainly that will be better than something from one of the major SI's because he will probably be using proper, off the shelf parts that anyone has access to. With a custom machine, you know exactly what every part in the system is, where you do not with a typical prebuilt.

Give this a watch, I think it might help illustrate what I am cautioning you about with regards to the prebuilts:

 

[Chaosrider]

Well you didn't share a budget So I picked a very mid-range system. It's a reference point if nothing else. A proper video editing workstation would be far more expensive than that.

Also as I mentioned, you don't need to build it yourself even if you are picking your own parts - there are very likely services in your area that will build you a custom PC, and it will be so much better than anything you find as a pre-built, especially from a big box store. There are online services that will do this for you as well.

Dell/Best Buy is perhaps the worst place to buy a desktop PC, for the reasons mentioned in my previous post. If you have a local guy who does this sort of thing, almost certainly that will be better than something from one of the major SI's because he will probably be using proper, off the shelf parts that anyone has access to. With a custom machine, you know exactly what every part in the system is, where you do not with a typical prebuilt.

Give this a watch, I think it might help illustrate what I am cautioning you about with regards to the prebuilts:

That video is extremely informative!

I forwarded your spec sheet to an email list that I run, and one of the guys said that if I got a Ryzen 5 3600, which comes with a Wraith cooler instead of a fan, it would serve my needs just as well, and be substantially cheaper.

Do you concur with that? What would I likely lose from that "downgrade" from your original spec?

As soon as I get the parts list finalized, I'm going to write it up, and put it out for bid to a couple of the local computer shops.

Thanks for your help!!




TCS