file copy soooooooo slow!

[vrdiver]

I've figured out a brand new, shiny backup regime*, which uses disk images (Macrium Reflect - free version) and Robocopy.

Macrium reflect allows me to create an image of a disk (or partition) which is great, but I want to keep incremental backups, which makes full disk images quite storage-intensive (one per day for the last 4 weeks).

I thought I'd write a script using the built-in Windows robocopy command: the script works fine, allowing me to back up specific folders and making full copies of the contents to my NAS, and then the next day making a copy of just the files that have been changed. Each week I end up with a "full" copy of the folders, and 7 "day" folders with incremental changes stored. in the event of mishap, I can go back to previous editions of a file for any day in the last 4 weeks (obviously you could choose how many days/ weeks to store, or whether to change the granularity to hours etc).

Unfortunately, robocopy seems to run excessively slowly between my PC and NAS (ethernet via router). I can use the XO switch to just update my "full" copy with latest changes, so the full bckup never has to be done from nothing after the first run, but I'd love to know how to speed up file copies across the network (music files currently being copied at < 5Mb/s).

VRD


*I'll post the backup regime separately as a sanity check on the approach.

[Infrasonic]

https://www.raymond.cc/blog/12-file-cop ... /view-all/

The comments section has some from experienced Robocopy/Rich Copy users.

[Infrasonic]

Multi threading has cropped up a few times as a way to speed up network (LAN) copies.

https://pureinfotech.com/robocopy-trans ... indows-10/

...Perhaps the most important switch to pay attention is /MT, which is a feature that enables Robocopy to copy files in multi-threaded mode. Typically, when you copy files using File Explorer, you’re only copying one file at a time, but with multi-threaded enabled, you can copy multiple files at the same time better utilizing the bandwidth and significantly speeding up the process...

[vrdiver]

https://www.raymond.cc/blog/12-file-copy-software-tested-for-fastest-transfer-speed/view-all/

The comments section has some from experienced Robocopy/Rich Copy users.

Unfortunately, nothing to address my speed issue (but thanks for the link)

I'm using the command*:

robocopy D:\data\ B:\Full\Data\ /MIR /S /J /R:3 /W:10 /DCOPY:T /XO /PURGE /MT:24 /V /NP /LOG:B:\Full\RoboBackup.log

inside a .bat file for the full backup.

D: is an internal HDD encrypted with bitlocker.
B: is a Veracrypt mounted container on the NAS

I suspect it's not robocopy specifically, as terracopy also runs slowly with the same source-destination files.

VRD


*Robocopy syntax available at https://www.computerhope.com/robocopy.htm

[Infrasonic]

As an A/B could you do a direct ethernet cable connection between the PC and the NAS to eliminate the router as a bottleneck issue?

I don't think you need an ethernet X-over cable these days, most modern kit should do it automatically (hopefully...)

Edit: Could the encryption be the issue slowing it down?

[vrdiver]

I'll try the direct NAS-PC connection, but it's a temporary solution due to location of kit .

Re encryption causing the issue: the acid test is, of course, to repeat the exercise without the encryption, but if that is the cause then I'm stuffed, as the "solution" is not great

[Infrasonic]

I'll try the direct NAS-PC connection, but it's a temporary solution due to location of kit .

Re encryption causing the issue: the acid test is, of course, to repeat the exercise without the encryption, but if that is the cause then I'm stuffed, as the "solution" is not great

If it is the encryption then running a SSD/SSHD at the source end and a SSD/SSHD cache at the destination (NAS) end could speed it up, as the latency will be a factor lower. Or have an experiment with RAM drives at either end (I've seen it done succesfully, but it seems to take a fair bit of tweaking to optimise. There's plenty of YouTube videos on it.)

4K random read/write speed is always an issue (which is why drive manufacturers always quote the much better sequential figures...), add in encryption on a mechanical HDD at either end and the fact that the DRAM cache will be small (128MB typically) and saturated pretty quickly and...

[Infrasonic]

As a cheap A/B if you've got USB sockets at both ends (PC-NAS) you could try running a couple of USB flash drives as source/destination with unencrypted data and see how fast that is?

Do it direct, then via the router, and then encrypted and see what the transfer bandwidth drop off is?

[Infrasonic]

Task Manager/Performance will give you the actual source HDD performance stats (as well as the ethernet bandwidth), so that should tell you straight away if it is a source disk 'read' issue.

[Slarti]

https://www.raymond.cc/blog/12-file-copy-software-tested-for-fastest-transfer-speed/view-all/

The comments section has some from experienced Robocopy/Rich Copy users.

Unfortunately, nothing to address my speed issue (but thanks for the link)

I'm using the command*:

robocopy D:\data\ B:\Full\Data\ /MIR /S /J /R:3 /W:10 /DCOPY:T /XO /PURGE /MT:24 /V /NP /LOG:B:\Full\RoboBackup.log

inside a .bat file for the full backup.

D: is an internal HDD encrypted with bitlocker.
B: is a Veracrypt mounted container on the NAS

I suspect it's not robocopy specifically, as terracopy also runs slowly with the same source-destination files.

VRD


*Robocopy syntax available at https://www.computerhope.com/robocopy.htm

How does a simple copy with Explorer compare for speed? As is drag and drop or Ctrl+C then Ctrl+V

Also, is xcopy still available in your version of Windows?

Slarti

[mc2fool]

I've figured out a brand new, shiny backup regime*, which uses disk images (Macrium Reflect - free version) and Robocopy.

Macrium reflect allows me to create an image of a disk (or partition) which is great, but I want to keep incremental backups, which makes full disk images quite storage-intensive (one per day for the last 4 weeks).

You don't need to keep lots of full disk images, Macrium Reflect Free does differential backups and Paragon Backup & Recovery Free does both incremental and differential backups, as indeed do some other free backup programs, e.g. EaseUS and AOMEI, so I'm not sure I see the need for faffing with robocopy -- and it will be a faff if you need to rebuild the disk.

I too have a (relatively) new backup regime, about a month old -- and I've already had to use it. I do weekly full image plus daily differential with Macrium Free and weekly full image plus daily incremental with Paragon Free on both my desktop and laptop, with all going directly to my NAS drive. Two regimes? Well, I was trying out a number of backup programs and got it down to a short list of those two, and just haven't chosen between them yet! And, actually, I don't have any real need to do so, so the redundancy is fine.

As it happens, my laptop HDD died a sudden death a few days ago, so I bought an SSD to replace it, booted the laptop from my Macrium Reflect recovery CD and restored the last differential (which automatically does the matching full image) from the NAS, rebooted and was up and away.

[vrdiver]

How does a simple copy with Explorer compare for speed? As is drag and drop or Ctrl+C then Ctrl+V

Also, is xcopy still available in your version of Windows?

copy and xcopy run at the same speed. After rebooting both the PC and the NAS, initial speeds were higher, but have slowed down again.

Task Manager/Performance will give you the actual source HDD performance stats (as well as the ethernet bandwidth), so that should tell you straight away if it is a source disk 'read' issue.

CPU is hardly used, HDD usage is floating around 1% (peaks at 10%) and ethernet was running at around 30 - 50Mbs but has now dropped back down to 15 - 20 Mbs.

Copy from the bitlocker encrypted drive to an unencrypted area on the NAS seemed to take a similar amount of time, so I can't blame the encryption. I've read that anti-virus software can slow things down but that's a price I'm willing to pay, rather than leave a backup exposed.

I've also increased the network adapter buffers to 2048 bits memory (send and receive) up from the default 512/256 settings. Tried setting NAS, router and PC to all use "Jumbo" packets, but again, doesn't seem to make a difference.

The NAS direct connection to the PC will take a bit more work (both have two ethernet ports, but the second cable (doesn't need to be a crossover cable, although I have one from when they were needed!) isn't being recognised despite setting static IP addresses and a DNS server name. In reality it's not a solution (for me, due to kit location and not wanting to run another cable) but I will play with it a bit more when I have some time.

Macrium Reflect Free does differential backups and Paragon Backup & Recovery Free does both incremental and differential backups

Thanks for that. I'd spotted the Macrium differential, but wanted the incremental option, so was experimenting with robocopy as I can configure that in a script. I hadn't discovered the Paragon Incremental option, so will download and have a play!

As for "faffing" with robocopy if it's needed for real - I'm a HYP investor, what else was i going to do with my time? (I have a restore script that would load up the full backup if younger than the macrium version and then apply the incremental backups to the point I wanted, or I would just copy over specific files. Macrium would have done the heavy lifting on the disk rebuild first though!

However - back to the drawing board and time to have a good look at the Macrium/Paragon options.

VRD

[mc2fool]

I have a restore script that would load up the full backup if younger than the macrium version and then apply the incremental backups to the point I wanted

How does it handle files deleted before the final incremental?

[mc2fool]

On your general speed issue, maybe an obvious suggestion, but have you tried another cable(s)? Also, what is Macrium reporting for the speed of the backups it's doing? (It's at the very bottom of the log).

As well as the direct PC-NAS test (and I wouldn't be trying with second cables/adapters as that introduces possible other issues, just the direct connection with just and only the "primary" ports on both), do you have another PC you can do a test with? Both for copies to/from the NAS, with and without router, and PC-router-PC, and PC-PC, as part of an elimination game.

I'd also be googling and/or looking the forums (if any) for both your router and your NAS to see if anyone else is having similar issues....

[vrdiver]

I shall have another play with speed issues when the current (full) backup is completed: never occurred to me to check if the cable was the culprit, but fair point - it's a quick test!

(current backup suggesting I will be twiddling my thumbs for the next 2 - 3 days, but I'd like to get one in the bag before continuing my quest for speed...)

VRD

[Slarti]

I shall have another play with speed issues when the current (full) backup is completed: never occurred to me to check if the cable was the culprit, but fair point - it's a quick test!

(current backup suggesting I will be twiddling my thumbs for the next 2 - 3 days, but I'd like to get one in the bag before continuing my quest for speed...)

VRD

Looking at your router, how many lights are on for the PC and how many for the NAS?

When I did my last full backup, on 1st, I thought that it was taking longer than usual, only to find that the PC was only running at 10mbps (only one light on at the router)

Also, I think that you have said that you rebooted PC and NAS, but did you reboot router?

Slarti

[Infrasonic]

Any joy yet?

Another possibility is to reduce the multithreading, the Robocopy default is 8 I think.
24 might be a bit steep if the hardware (HDD) isn't up to it, especially as you'll have the encryption and AV to deal with?

Have you tried whitelisting in the AV just to see if it makes any difference to throughput? (I understand you don't want to turn it off, but just as an A/B to trace the potential causes of the issue.)

[vrdiver]

Hurrah! a full back-up finally in the bag.... Only taken about 5 days

Changed the cables - no impact
Connected NAS directly to PC - no impact (rebooted both and ran file copies from PC to NAS)
Told Windows Defender to Exclude NAS - no impact
Turned off AV - slightly faster initial transfer, then dropped back to slow-as-normal
Changed robocopy threads (tried 8, 12, 16, 24) - no impact (although 16 seemed slightly faster to start with)

NAS is a ReadyNAS Ultra 6 with 4 x 3Tb WD Red disks in RAID5 configuration
Router is an Asus RT-AC66U happily running at Gigabit speeds (I set the ethernet adapters to run at 1 Gbps full duplex on both PC and NAS, but as made no difference, changed back to auto).

Loaded a netdata app to monitor NAS RAM, disk usage etc.
CPU did spike to 100% occasionally but is not held there, so doesn't seem to be the root cause.
RAM (1Gb DDR2) is running at about 60% used CPU at around 30%, IPv4 at about 15Mb/s (~2MB/s).

Netdata monitor flashed critical warnings on RAM and swap space just as the copy rate plummeted

Have ordered 4Gb of RAM to replace the current 1Gb (£10, so hardly a deal breaker if it isn't the prime issue). Not sure if the NAS will be able to address 4Gb or only 3Gb, but we'll find out. The unit is out of warranty, so there are no issues with me "voiding" the warranty, only with actually breaking something!

in the meantime, googling has come up with plenty of posts with the same (slow copying) issue, but no applicable solution, as far as I can find.

Next steps -
#1 play with Paragon to see if it is a better solution than my custom script and decide if Custom Backup, Macrium or Paragon is the way forward
#2 Set up off-site backup regime (will be a separate post).

VRD

[Infrasonic]

http://www.storagereview.com/netgear_re ... w_rndu6000

I've just read all the way through this, some of the 4K random read throughputs are very low (even lower than you are getting!), which might explain the issues you're having.
They tested it RAID 5 as well.

Another A/B.
Have you got any large audio or video files that you could try, as they'll be sequential and should be much faster to transfer.
At least it will prove whether it's the file size or something more fundamental causing the issues.

Is there any way of using a spare bay as a cache? If yes you could put an SSD in (with a 3.5"-2.5" adapter) and run that to reduce the latency and improve the IOPS.

[vrdiver]

http://www.storagereview.com/netgear_readynas_ultra_6_review_rndu6000

I've just read all the way through this, some of the 4K random read throughputs are very low (even lower than you are getting!), which might explain the issues you're having.
They tested it RAID 5 as well.

Another A/B.
Have you got any large audio or video files that you could try, as they'll be sequential and should be much faster to transfer.
At least it will prove whether it's the file size or something more fundamental causing the issues.

Is there any way of using a spare bay as a cache? If yes you could put an SSD in (with a 3.5"-2.5" adapter) and run that to reduce the latency and improve the IOPS.

I did some more testing, using a randomly selected folder of files, and a couple of large video files:



SSD caching is in the developers "soon" list: I'll keep an eye out for when the next firmware release comes out. I have two of the six NAS bays still available, so will be interesting to see if it becomes an option.