What is the rationale behind safely removing USB media from windows PCs?

Question

Quite often when I'm in a rush, I automatically pull out a USB pendrive or USB cable from an external hard drive from my desktop PC or laptop, without right-clicking on the safely remove icon in the system tray and unplugging via this route. So far nothing untoward has happened every time I have "been in a rush".

What is the rationale behind right clicking on the safely remove icon and can I really lose information on USB media if this is not carried out?

Edit

Is the vunerability of losing such information greatly increased, if the usb media is still flashing at the time of pulling it out of the computer (as opposed to non-flashing)?

score 104 · Accepted Answer · edited Sep 20 '13 at 07:40

Yes it can, it's about what happens if you remove the device when it's in use (reading or writing).

When you plug in a USB drive, you give your PC free rein to write and read data from it; some of which is cached.

Caching occurs by not writing information immediately to the USB device, and instead keeping it in your PC's memory (RAM). If you were to yank the USB drive out of your PC before this information is written, or while its being written, you'll end up with a corrupted file.

However, Windows automatically disables caching on USB devices, unless you specifically say that you want it enabled. For the most part you don't have to click the 'Safely Remove Hardware' button, if you aren't writing or reading anything from the device.

Its there simply as an extra level of security preventing you from destroying your own files.

Doing so causes the files to close "gracefully", preserving data, pointers and file size indicators. When writing to disk the computer doesn't always "flush" a buffer and only part of the data may have been written. Using the proper procedure will assure that the data and pointers are in good shape.

Source

score 45 · Answer 2 · answered Sep 18 '13 at 13:46

A second reason is that flash drives need to have stable power for ~0.25 seconds after a write command. This is a fundamental physical problem, due random factors some writes may leave a logical 1 bit in a electrical 0.72 state. The fix is easy: just rewrite the bit, perhaps even a few times. Eventually it will stick.

If you're really unlucky, the bit falling over will be in a filesystem table and corrupt e.g. an entire directory.

score 21 · Answer 3 · answered Sep 18 '13 at 16:43

Few people realize everything that goes on under the hood of a flash drive. Unlike hard drives in which a sector is erased and overwritten in a single step, flash drives can write data to a blank 528-byte page much faster than they can perform an erase cycle. Additionally, since each erase-control circuit would use up a fair amount of silicon, flash chips generally don't have a separate circuit for each page but instead have one per "block", where each block contains hundreds or thousands of pages. This means that every time different data is written to a sector, it will be written to a previously-blank area and the old copy marked somehow as "out-of-date". At some point, the flash will perform a "garbage-collection" cycle where it will identify a block that contains mostly out-of-date pages, copy all pages that aren't out of date from that block to another block which has space, and then erase the block. Because such operations may be very slow, some drives will attempt to, when possible, perform them when nothing else was going on; drives don't tell the computer when they perform such cleanup, because the computer generally won't care. The "gotcha" is that if the computer doesn't let the drive know that it's about to lose power, the drive will have no way of knowing when it's safe to perform such cleanup.

Note that although quality flash drives are designed to ensure that a new copy of data will always be written completely before an old copy is erased, and ensure that they can always figure out which copy of each block should be considered the most current, not all flash-drive designs are perfect. In the absence of any power-fail notification, an algorithm which is designed to be 100% proof against data loss or inconsistency would probably be at least twice as slow as one meeting looser constraints. Adding a hardware power-fail notification which would trip while there was still enough energy to complete one or two flash-write operation would reduce the time penalty, but making power-fail notifications 100% reliable can be tricky, and not all implementations are perfect.

Augustus Francis · Answer 4 · 2013-09-21T09:14:56.930

Windows itself tells you that you don’t need to use the Safely Remove Hardware option if you use certain settings – the default settings.

Quick Removal vs. Better Performance

Windows allows you to optimize your USB device for quick removal or improved performance. By default, Windows optimizes USB devices for quick removal. You can access this setting from the device manager – open the Start menu, type Device Manager, and press Enter to launch it.

Expand the Disk drives section in the Device Manager, right-click your device, and select Properties.

Select the Policies tab in the Properties window. You’ll notice that Windows says you can disconnect your USB device safely without using the Safely Remove Hardware notification icon, so this means you can unplug your USB device without ever safely removing it, right? Not so fast.

Data Corruption Danger

The Windows dialog shown above is misleading. If you unplug your USB device while data is being written to it – for example, while you’re moving files to it or while you’re saving a file to it – this can result in data corruption. No matter which option you use, you should ensure that your USB device isn’t in-use before unplugging it – some USB sticks may have lights on them that blink while they’re being used.

However, even if the USB device doesn’t appear to be in-use, it may still be in-use. A program in the background may be writing to the drive – so data corruption could result if you unplugged the drive. If your USB stick doesn’t appear to be in-use, you can probably unplug it without any data corruption occurring – however, to be safe, it’s still a good idea to use the Safely Remove Hardware option. When you eject a device, Windows will tell you when it’s safe to remove – ensuring all programs are done with it. Write Caching

If you select the Better Performance option, Windows will cache data instead of writing it to the USB device immediately. This will improve your device’s performance – however, data corruption is much more likely to occur if you unplug the USB device without using the Safely Remove Hardware option. If caching is enabled, Windows won’t write the data to your USB device immediately – even if the data appears to have been written to the device and all file progress dialogs are closed, the data may just be cached on your system.

When you eject a device, Windows will flush the write cache to the disk, ensuring all necessary changes are made before notifying you when it’s safe to remove the drive. Ultimately, no matter which option you use, you should use the Safely Remove Hardware icon and eject your device before unplugging it. You can also right-click it in the Computer window and select Eject. Windows will tell you when it’s safe to remove the device, eliminating any changes of data corruption.

Note This advice doesn’t just apply to Windows – if you’re using Linux, you should use the Eject option in your file manager before unplugging a USB device, too. The same goes for Mac OS X.

acarbonaro · Answer 5 · 2014-02-27T08:28:36.003

Failing to safely remove your devices causes phantom devices to remain in Windows. This isn't usually an immediate problem, but once you've amassed a few hundred phantom devices things start to go wrong.

EDIT FOR CLARIFICATION: Phantom devices are, in short, devices that have drivers installed but are no longer connected to the USB port. Windows keeps track of devices that have been installed, but because the devices hasn't been "safely removed" the driver is still loaded for that USB port. This can cause a variety of problems ranging from assigning a new drive letter when you plug in your flash drive to causing video games to detect controllers that aren't there. When you safely remove a device from Windows you're not just making sure it isn't writing data anymore, you're telling Windows to unload the driver from that USB port. Unplugging your drive on the fly not only risks data corruption, but also risks creating a phantom device that Windows doesn't really handle well.

By default, Windows doesn't show phantom devices in Device Manager, which is why most people are unfamiliar with the concept. To find these devices manually run Command Prompt as an Administrator and enter set devmgr_show_nonpresent_devices=1. This will allow Device Manager to show the phantom devices for the current session.

It is quite possible that there are many too many phantom devices to easily remove them all by hand. GhostBuster is a lightweight GUI designed to easily display and remove phantom devices.

Reference: https://windowssecrets.com/newsletter/how-to-prevent-and-remove-phantom-devices/

score 6 · Answer 6 · answered Sep 19 '13 at 14:45

In general, when the computer is writing to the USB drive, you run the risk of corrupting the file system or files when you pull it out without safely removing it.

FAT32

Then, when you have a FAT32 disk using Windows' default settings, Windows won't cache the writes so you can yank the drive out when it is not being written to.

However, you can improve FAT32 performance by enabling write caching. However, then you must safely remove the drive or otherwise not all data is written to the disk.

enter image description here

NTFS

And when you use NTFS, since it is a journaling file system, you can't break the file system by yanking out the drive. However, you will lose or corrupt files if you do that.

Surprise removal doesn't result in corruption, at least from NTFS's point of view. The file system data structures remain intact (or at least can be repaired from the change journal) regardless of when you yank the drive out of the computer. So from the file system's point of view, the answer is "Go ahead, yank the drive any time you want!"

Since NTFS is a journaling file system, it can auto-repair the internal data structures that are used to keep track of files, so the drive itself remains logically consistent. The file is correctly set to the final size, and its directory entry is properly linked in. But the data you wrote to the file? It never made it.

Even though NTFS is robust and can recover from the surprise removal, that robustness does not extend to the internal consistency of the data you lost.

-- Raymond Chen

score 3 · Answer 7 · answered Sep 19 '13 at 10:48

Have you ever noticed that sometimes, when "safely" ejecting a USB Flash Drive, it takes several seconds until you're told it's safe to remove it? And that during this time, the disk keeps flashing its activity light?

I do see this often, and that alone is indication enough for me not to risk yanking out a disk even when it appears never to lose any data when I yank it out accidentally sometimes.

A bit more background on what may be going on:

Possible disk corruption.

Others have already explained them pretty well, especially the one by supercat is pretty important for flash drives.

Possible program issues.

This is where programs that have files open on the disk are not well prepared to handle the case where the files suddenly become inaccessible. These programs might crash or freeze up then, depending on how well they were developed.

Therefore, by using the "remove hardware safely" command, you start a process in the operating system that asks these programs to let go of the disk. Once all programs have let go if it, you are told that you can now safely disconnect the disk.

However, if there's a program running that does not let go of the disk, then you will be told so as well, and then you have the option to close that program so that it stops using the disk, preventing both data corruption on the disk and program issues on your computer.

Even more background on caching and disk corruption

A disk that stores files works like a library with indexes. The challenge is that the index needs to be kept in sync with the books (files) it contains. When you add/remove a file on a disk, its index (i.e. the directory) needs to be updated as well. All this happens in several steps, and yanking the disk out in the middle of it leads to corrupted disk structures (to identify and repair them, tools such as "chkdisk" come to the rescue).

Howevever, keeping the directories up-to-date all the time requires a lot of writing and time. So, if you write 100 files to the disk, the operating system (e.g. Windows) does not write each file followed by an update of the (often same) directory, but optimizes this operation by first writing the data of the 100 files to disk, and only then write the updated directory in one go.

Such an optimization makes copying (and deleting) files much faster, and modern operating systems tend to delay the update of the directory as much as possible to speed the computer up even more.

I.e, the operating system keeps the correct directory only in its local memory (RAM or internal hard disk, which is usually still faster then the USB connection). This is the caching that others speak of: Keeping data in a separate location for faster access.

But what if the entire computer has a power failure? Well, even then it has ways to recover from it as soon as it gets powered on again, keeping the directory intact (using "journalling", for example). Of course, that only works if the disk whose directory was cached is still present after power comes back - which is usually the case, so nothing to worry here.

But all these safeguards can't work if you simply remove the disk and do not give the operating system to write its cached directory back to disk, obviously.

Therefore, you announce to the operating system (and the programs it runs) that you intend to remove the disk. This process, also called "unmounting", leads to writing all cached data to the disk and then stopping further access to it.

score 1 · Answer 8 · answered Sep 18 '13 at 15:01

Flash is only able to write one's and no zeros. It's able to erase a whole block (setting it to zeros). Because most flash media uses Wear Leveling it only adds a block to a "to be deleted" list and uses another block instead. The controller on your usb media begins erasing blocks on the "to be deleted" list whenever nothing else has to be done. Plugging the power while blocks are being erased may let them in a undefined state.

score 1 · Answer 9 · answered Dec 19 '13 at 21:59

What is the rationale behind safely removing USB media from windows PCs?

This function is for removable media. That is, media that is connected via USB, FireWire, and as such, can be disconnected at any given moment.

Accessing the drive to write every piece of data to it as requested would put a lot more burden on the drive which would reduce performance and increase wear. To increase performance and reduce wear and tear, Windows can cache data in order to write a bunch of data at a time. It’s like delaying a trip to the other end of town until you have enough errands to do there to make the trip worthwhile.

The problem with caching is that if the drive happens to be unplugged between the time that the write request was made and the time that Windows “flushes” the cache to disk (actually performs the write), then that data would not get written. This leads to data loss and possibly even corruption.

By “ejecting” a removable volume, you tell Windows to flush the cache and write any data it has waiting. That way you ensure that it is safe to unplug the drive.

Quite often when I'm in a rush, I automatically pull out a USB pendrive or USB cable from an external hard drive from my desktop PC or laptop, without right-clicking on the safely remove icon in the system tray and unplugging via this route. So far nothing untoward has happened every time I have "been in a rush".

That’s mostly because Microsoft is not stupid (lets not argue here…) They know that people don’t like doing extra work. Therefore, when you plug a removable media into the system for the first time, when Windows installs drivers for it, part of the configuration it does is to set it for “quick removal”.

In the Device Manager, in the Policies tab of the Properties dialog for your drives, depending on the type of drive, you can choose to configure the drive for either quick-removal or better-performance or enable or disable write caching.

Enabling write-caching or selecting better-performance will cause Windows to queue write requests which increases performance and reduces wear, but also increases chances of data-loss and requires ejecting the volume before removing it. Disabling write-caching or selecting quick-removal causes Windows to write everything to the disk immediately which reduces performance and increases wear, but makes it okay to simply pull the drive out.

Because Windows defaults removable media to quick-removal, most people will rarely experience a problem from just unplugging it without ejecting first.

The screenshots below show the default drive policies for a variety of media. There are few interesting points to notice.

The internal hard-drive is set to use write-caching (figure 1). This makes sense because it is assumed that the drive is a permanent fixture, so it will never normally become unavailable in the short period between a write request and a cache flushing.

The USB flash-drive is set for quick-removal (figure 2). This too makes sense because it makes it convenient to remove it easily.

The USB memory-card is configured for quick-removal (figure 3). This is logical because it is essentially no different than a USB flash-drive (or any other USB drive). What’s interesting is that the internal memory-card reader is also set to quick-removal (figure 4). The reader itself is built into the laptop, and therefore always present, but the card can be removed, and so it defaults to quick-removal. (This is the same for floppy drives; there’s no caching because the disk in the drive could be removed.)

Even more interesting are the external USB hard-drive (figure 5), and iPod mini (figure 6) which has a little, spinning hard-drive in it. These are actual hard-drives, not flash-drives, but because they are removable, they still default to quick-removal. Again, this makes sense because they can be unplugged before the data is written to the disk.

Can I really lose information on USB media if this is not carried out ?

Is the vunerability [sic] of losing such information greatly increased, if the usb media is still flashing at the time of pulling it out of the computer (as opposed to non-flashing) ?

Yes. The problem is that even if a drive is set to quick-removal does not make it safe to unplug it right away. Just because Windows performs the writes immediately, does not mean that they occur immediately.

Most removable media such as flash-drives and memory-cards are much slower than hard-drives. Therefore, even though Windows initiates the write as soon as it is requested, it will usually take a few moments for the data to actually get written. Even if the write/copy/etc. dialog box that shows the write progress goes away, the drive may still be writing for a few more moments, and if it has its own drive-activity LED, you can see this. It is not safe to remove the media right away; watch the LED and wait for it completely stop flashing.

To be extra safe, you can use a program like Sync tell Windows to flush the cache. Essentially, it is exactly the same as using the safely-eject function, but can be a little more convenient if you make a shortcut on the desktop to run it with the specific drive-letter of your removable drive. It will save about 5-10 seconds which doesn’t sound like much, but in HCI, is a huge barrier. What’s even better is that it only flushes the cache, it does not actually eject the drive. Therefore, the drive is still mounted and available for use if you suddenly remember something you forgot to do with it. This makes it even more useful because you can even assign it a hotkey so that you can flush the cache and make it safe for unplugging with a keystroke.

Figure 1: Internal HDD policy

Internal HDD policy