When copying a large amount of data, 200GB, nominally all held in one folder but divided amongst subfolders, from one HDD to another ..........

[Yorkshire_Pud]

is it better to copy individual sub folders one at a time ? Though this would be laborious

I ask because the crurrent copy is just copying the 'nominal' folder and as I am watch the progress and it seems to be copying a 'random' sequence of files.
This is making me think it is working from the 'outside' of the original HDD to its 'centre' and might mean the the copied data is similarily arranged on the recieving HDD.

 

[GFields]

Not sure what you mean by "nominal" but most likely, if you're using Windows, go into options and make sure "view all files/folder" is selected. Then try again. Or use a migration application.

 

[Yorkshire_Pud]

Nominal ? in my usage meant, one folder e.g. folder A, that contain all the other folder, some of which are in 'trees'.
So, folder A contains folders B to Z, and each of those folders contain other folders and each of tho.......... ad -infinitum,

I am copying "folder A' from one HDD to the other.

 

[GFields]

Nominal ? in my usage meant, one folder e.g. folder A, that contain all the other folder, some of which are in 'trees'.
So, folder A contains folders B to Z, and each of those folders contain other folders and each of tho.......... ad -infinitum,

I am copying "folder A' from one HDD to the other.

You can do that in Windows Explorer but as I mentioned, make sure to make the view changes. 200 GB will take a while is why I mentioned a migration application. But if this is something you only do every now and then, using Windows Explorer won't cost you anything. Data transfer is one of the services I offer with my sister company is why I have the application.

 

[Yorkshire_Pud]

You can do that in Windows Explorer but as I mentioned, make sure to make the view changes.

I am,

200 GB will take a while is why I mentioned a migration application

It IS LOL, windows original estimate was 23 hours, but, with 2 hours elapsed, it has dropped to 2 hours with 58% remaining. It remains to be seen if that estimate is accurate lol.

 

[djiuser12345]

It depends on what filesystems are in use and what files you copy.

If each file (in whatever folder) is the same size 5gb the operating system might chose to copy them sequentially. Otherwise its very likely that it starts somewhere (folder A/K/E) and tries to copy 3 small files from that folder while also copying 2 big files from a different folder or at least preparing the copy for them.

A lot of variables in play here

 

[Yorkshire_Pud]

It depends on what filesystems are in use and what files you copy.

The defaults win 7 file system, file sizes? from a few mb to 10s of GB.

Otherwise its very likely that it starts somewhere (folder A/K/E) and tries to copy 3 small files from that folder while also copying 2 big files from a different folder or at least preparing the copy for them.

That looks to be what it is doing, it has copied 18,000 files with 4,500 keft and the files currently being copied are large DJI export DATs from mavic style drones, I should have binned them, and each is taking quite a bit of time.

I am just wondering if, for 'fragmentation'-of-a-folder putposes, it would have been better to manually copy them to the second HDD folder by folder.

 

[djiuser12345]

Small files are problematic for speedy transfers.
Large files can be copied like a stream at full bandwidth, smaller ones create much more overhead which impacts speeds

 

[Yorkshire_Pud]

48 % remaining, ho hum. Transfer rate creeping up,

3 hours lapsed and 30% remaining

 

[VenomXts]

Lot of variables, few programs I would use to take the modified dates with the copy personally depending on what your doing. 200 gigs isn't a lot of data anymore but how your moving from to what can make all the difference. I take 200 gigs of data off a sdcard occasionally if I got lazy. I moved 15 terabytes recently as well and that did take awhile. If it's windows just make sure you copy and don't cut and your usually ok.... if you want to be safe though having.something that checks the hash # is the way to go IMO.

 

[Yorkshire_Pud]

that checks the hash #

Thanks but, uuum, what does that mean ?
I have to crank my computer with a clockwork handle .... it is that old lol and the connection was via two USB cables which probably didn't help.

 

[VenomXts]

Thanks but, uuum, what does that mean ?
I have to crank my computer with a clockwork handle .... it is that old lol and the connection was via two USB cables which probably didn't help.

Md5 hash is a unique series of numbers and letters created for a file that is unique to the data present in that file. There are a number of ways to look it up but simply if you copy a file and it has the same md5 hash as the original nothing changed if it did then you may have flipped a bit or something happened to it and its a way to verify the integrity of copy.

Yah, old usb protocols 2 vs 3 etc have diffrent speeds 200 gigs on older USB protocols can take a lot of time!

I can't remember the free programs name i use to copy and check.the md5 hash and am away from my computer but will post name later if your intrested.

Hope that explains a little more though.

 

[djiuser12345]

Teracopy is alright for windows, can do checksum calculation for you.

If you have some half decent drives amd good connections you usualy don‘t need to worry about it

 

[Dangerly]

On Windows you have faster ways of transferring large amounts of data than using the built in file explorer drag and drop:

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

In Windows, you can transfer 100s of GBs with one drag and drop, and the amount of time it takes is of course proportional to the amount of data you are transmitting, but also is largely dependent on the kind of hard disks you have and how they're connected to your computer. For example, SSDs are faster than HDDs, and USB 3.0 is way faster than USB 2.0.

The time it takes to transfer could vary by 10x or more depending on your setup.

Estimated transfer times will be wildly off at the beginning.

 

[Kilrah]

Sounds like you're using an old USB2 connection - that's going to be the limitation. Copying folders one by one would gain nothing.

 

[Yorkshire_Pud]

Just to clarify a point, speed was not / is not my concern, it takes whatever time it takes.

The concern was and is that, going by the sequence of tranfers that I saw, data from any given folder could be spread all over the new HDD.
In order to write the data from one given folder to one area of the new HDD I was/am wondering if it would have been better to copy one subfolder at a time ?

 

[rwilliam99]

Just to clarify a point, speed was not / is not my concern, it takes whatever time it takes.

The concern was and is that, going by the sequence of tranfers that I saw, data from any given folder could be spread all over the new HDD.
In order to write the data from one given folder to one area of the new HDD I was/am wondering if it would have been better to copy one subfolder at a time ?

You cannot (and really do not want to even attempt) to control where the data will be written physically on the HDD. Years ago, it was much more of an issue, but today, I would say it is WAY less of a concern.

If you really want to try and optimize it, you can always defragment the HD after you do the copy.