I work with tens of thousands 20 kB JPEG files in batches which add up to gigabytes. When I try to delete an old batch, Windows 10 has a very low delete rate which sometimes even drops to zero for some reason:
Is this just a fact of life for the hardware I'm on, or is there some tips and tricks I could try to speed up this deletion process?
From the image it looks like you are deleting the files through Explorer, which is the slowest method possible.
What you can do to improve:
Delete the files using Shift+Del so the deleted files are not moved to the Recycle Bin (no recovery possible)
Issue the delete from inside a Command prompt using a command
similar to (use del /? to see all parameters):
del /f /q *.* (del in current folder, add `/s` to traverse sub-folders)
del /f /q /s folder
I don't know why Explorer is so slow, but you might consider putting each batch of images on a separate filesystem volume and quick-formatting the volume to delete them. Windows 10 has native support for mounting files as disk volumes. The speed of reformatting is independent of what's on the volume since it ignores the existing file system and just writes a new one.
Tens of thousands of 20K files sounds like not too many gigabytes. If the total size of all live batches is smaller than your physical RAM, you could also put them on RAM disks (using, e.g., ImDisk), still formatting or unmounting the volume to delete them. Even ordinary recursive deletion on the RAM disk might be much faster depending on what's causing the slowdown. This is obviously not an option if you need the files to survive a reboot.
One disadvantage of reformatting to delete is that you need to be an elevated administrator. Another is that you can accidentally format the wrong volume, but really any approach to mass deletion of files carries the risk of deleting the wrong ones. If you give the volumes descriptive labels like TempJpeg1 then formatting them is probably as mistake-proof as deleting a huge directory from the command line.
Disable any antivirus/antimalware real-time protection you have, at least for the duration of this operation. This includes Windows Defender. Also any anti-ransomware protection (as offered by some backup programs).
Real-time protection typically has hooks on file operations, and are historically notorious for drastically slowing down any operation involving large numbers of files.
If you feel the need to, you can re-enable them after the deletion is complete.
Running this as a command will be a lot faster. Launch cmd and run
RD frames /S /Q
That will remove the folder and all its files and subdirectories in it, without spamming the screen with each file name it is deleting.
RD = remove directory
/S = subdirectories included
/Q = quiet mode
Dealing with roaming profiles, I've seen this quite a bit. In my experience, sending any sort of delete command is not as fast as using Robocopy to /MIR an empty folder to the destination. I don't have any raw data or actual metrics that this is faster than any of the already mentioned options, but it is considerably faster than using the Explorer shell.
Just make a new empty directory, or reference a current empty directory, then run the following command:
robocopy x:\path\to\emptyfolder x:\path\to\delete /MIR /R:1 /W:1 /MT:128 /LOG:x:\path\to\logfile.log
/MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).
/R:1 :: number of Retries on failed copies: default 1 million.
/W:1 :: Wait time between retries: default is 30 seconds.
/MT:128 :: Do multi-threaded copies with n threads (default 8).
n must be at least 1 and not greater than 128.
This option is incompatible with the /IPG and /EFSRAW options.
Redirect output using /LOG option for better performance.
You'll want to LOG when using MT, there is definitely a measurable delay when displaying the output on screen.
By using /MT:128 as an option, it will max out the amount of files it can 'delete' at one time.
Also note that if this is your first time with Robocopy, you should do a dry run with the /L flag:
/L :: List only - don't copy, timestamp or delete any files.
Then check the logs and ensure the actions match what you want.
References:
Create a partition per batch, let's say 10GB or so, when done with given batch quick format the partition. All files gone in matter of seconds.
To add to the answers provided, the reason that the deletion operations take so long is due the type of hard drive you're using. In one of your comments you state that you're using a mechanical hard drive, and the amount of IOPS that they can perform is pretty small compared to SSDs. The Wikipedia has a list of the rough amount of IOPS that mechanical and solid state drives can perform on their site.
What's basically happening is that the OS is telling the hard drive to erase the metadata of the file in the partition table, which removes it from view, (it's still there, but the OS no longer knows how to access it). It's basically a very small write to the hard drive, (it's writing 0s to the it), so it's a quick IOP, which can do a lot more than what's listed on the Wiki since the Wiki is listing random and sustained writes.
The thing to keep in mind now is that Explorer tries to play nice with everything else running on the system while it is deleting, which can be slow. The del command on the other hand doesn't try to be nice at all, and deletes as quickly as possible.
Upgrading to an SSD would help with your work since their IOPS are normally measured in the thousands, and mechanical drives are measured in the hundreds. You could also try using a flash drive to see if that helps; however your results would be highly dependent on the drive you use. If you have the RAM, (and don't mind the risk of loosing the images), you could create a RAM drive and use that. I have personally use ImDisk Virtual Disk Driver, (no affiliation), with a great amount of success and it's free.
I had this exact problem with some automated testing that created and deleted thousands of files for each test run.
Here is what I did to get good performance.
Always delete the entire folder and avoid the recycling bin. If you are doing it manually, then use the command line.
Can you convert some of your memory to a RAM drive? I bought extra RAM and it worked brilliantly because the files were supposed to be deleted as part of my testing. (A better solution is dedicated battery backed RAM drives, but they are very expensive.)
Writing to 1st and 2nd generation SSD drives gets slower over time and they eventually become unusable. Running a low level "Trim command" fixes this, although not all OS/Hardware combinations support Trim. Eventually I needed to buy new SSDs because mine had been rewritten too many times. (Obviously my testing was was killing my cheap SSD drives. Took me a while to figure that out.)
If you are using a HDD, defragment the drive regularly.
Good luck. It took me many months and a few SSDs, followed by extra RAM, to find a strategy that worked for me, so I feel your pain..
If you're deleting many small files the disk has to spend a lot of seeking, especially since you're using a spinning disk drive and not an SSD. Deleting a big file is like running a marathon all at once, but deleting small files is like running a marathon in 100 meter chunks spread out across the city. You're main activity is seeking the location in the disk. If this is a regular activity you could make a partition for holding these files temporarily and then wipe the partition in a second when you're done with them.
Besides the other great answers I wanted to share a few tricks with you:
del *.*del..Now make your life easier:
echo Y | del.
Deleting many directory entries will take longer than deleting a single directory of the net size. When deleting many files, it takes many commands instead of a single when a single file is deleted.
Now, when one big file is deleted, the filemapper is told to unallocate the memory of that big file, but the 1s and 0s of the file are not reset. The mapper is just told it is free to write new stuff over the old 1s and 0s. Basically, it is one command, that takes time to clear up all of the memory allocations of the fragmented file. When you delete a million small files, you are sending a million commands to the mapper, that has to do each one at a time.
So the solution is to use software to wipe a drive of deleted files and the 1s and 0s left behind. It will take one single command to wipe the space. Use the software only if you think the files are not needed to be recovered later.
After none of the other answers worked for me, I realized my problem was a 3rd-party tool: TortoiseGit.
It seems that when many files are deleted, TortoiseGit works furiously to update icon overlay information for the files being deleted (indicated by high CPU utilization by TGitCache.exe during file deletion).
I found a couple of solutions:
rm -rf in the Git Bash terminal seems to sidestep any involvement from TGitCache.exe.
I guess a similar problem may exist for other 3rd-party tools that generate icon overlays.
