Why is “/dev/rdisk” about 20 times faster than “/dev/disk” in Mac OS X

Question

According to the rasbery pi documentation, You can load your OS to a flash card with either /dev/disk or /dev/rdisk.

rdisk stands for raw disk.

/dev/disk is a block level device, why would rdisk be 20 times faster?

Using Mac OSX

Note: In OS X each disk may have two path references in /dev: /dev/disk# is a buffered device, which means any data being sent undergoes extra processing. /dev/rdisk# is a raw path, which is much faster, and perfectly OK when using the dd program. On a Class 4 SD card the difference was around 20 times faster using the rdisk path.

Answer 1

The accepted answer is right, but it doesn’t go into much detail.

One of the key differences between /dev/disk and /dev/rdisk, when you access them from user space, is that /dev/disk is buffered. The read/write path for /dev/disk breaks up the I/O into PAGE_SIZE chunks (4KB on x86 and arm, 16KB on arm64), which it reads into the buffer cache, and then copies into the user space buffer (and then issues the next chunk-sized read…). This is nice in that you can do unaligned reads and writes, and it just works. In contrast, /dev/rdisk basically just passes the read or write straight to the device, which means the start and end of the I/O need to be aligned on sector boundaries.

If you do a read or write larger than one sector to /dev/rdisk, that request will be passed straight through. The lower layers may break it up (eg., USB breaks it up into 128KB pieces due to the maximum payload size in the USB protocol), but you generally can get bigger and more efficient I/Os. When streaming, like via dd, 128KB to 1MB are pretty good sizes to get near-optimal performance on current non-RAID hardware.

The caching being done by /dev/disk’s read and write paths is very simple and almost brain dead. It caches even if not strictly necessary; like if the device could memory map and directly transfer into your app's buffer. It does small (4KB/16KB) I/Os, which leads to a lot of per-I/O overhead. It does not do any read ahead or write behind.

Answer 2

From man hdiutil:

/dev/rdisk nodes are character-special devices, but are "raw" in the BSD sense and force block-aligned I/O. They are closer to the physical disk than the buffer cache. /dev/disk nodes, on the other hand, are buffered block-special devices and are used primarily by the kernel's filesystem code.

In layman's terms /dev/rdisk goes almost directly to disk and /dev/disk goes via a longer more expensive route

Answer 3

It seems /dev/disk and /dev/rdisk works different for HDDs and SSDs. Wanna checked it for MicroSD card. Just wrote 2GB disk image to Sandisk Ultra MicroSD 64GB (https://www.amazon.com/gp/product/B073JYVKNX).

Repeated tests several times, but results were stable: 17MB/s for /dev/disk vs 20MB/s for /dev/rdisk. Changing bs=1m to bs=16m gives absolutely no difference in writing speed.

1. Writing to /dev/disk2

   sudo dd if=~/Downloads/ubuntu-18.04-4.14-minimal-odroid-xu4-20180531.img of=/dev/disk2 bs=1m
   2094006272 bytes transferred in 121.860007 secs (17183704 bytes/sec)
   

2. Writing to /dev/rdisk2

   $ sudo dd if=~/Downloads/ubuntu-18.04-4.14-minimal-odroid-xu4-20180531.img of=/dev/rdisk2 bs=1m
   2094006272 bytes transferred in 102.743870 secs (20380839 bytes/sec)
   

Then I decided to test reading speed: 26MB/s for /dev/disk vs 87MB/s for /dev/rdisk. Changing bs=1m to bs=16m gives absolutely no difference in reading speed.

1. Reading from /dev/disk2

   sudo dd if=/dev/disk2 of=~/Downloads/ubuntu-18.04-4.14-minimal-odroid-xu4-20180531-2.img bs=1m
   257949696 bytes transferred in 9.895572 secs (26067184 bytes/sec)
   

2. Reading from /dev/rdisk2

   $ sudo dd if=/dev/rdisk2 of=~/Downloads/ubuntu-18.04-4.14-minimal-odroid-xu4-20180531.img bs=1m
   877658112 bytes transferred in 10.021974 secs (87573377 bytes/sec)
   

Answer 4

I know that this is an old thread, but other folks might be interested in the speed implications of what I've tried. I want to backup my internal SSD in my MacBook Pro 13" Retina (with a Silicon Power 1 TB SSD) to an external USB 3.0 2.5" hard disk drive, wanting to capture both the macOS and BOOTCAMP partitions. My initial command line was:

sudo dd if=/dev/disk0 of=/dev/disk2 bs=1m

The results were a copy rate of ~31.3 MB/second. This was just too long to keep me waiting. So, on the second attempt, the command line was:

sudo dd if=/dev/rdisk0 of=/dev/rdisk2 bs=1m

Using /dev/rdisk instead of /dev/disk significantly sped things up, to about 98.4 MB/second! However, it gets even better. So, for the third attempt, I used this command line:

sudo dd if=/dev/rdisk0 of=/dev/rdisk2 bs=1m conv=sparse

The sparse option tells DD not to bother writing to output blocks that are all 0s on the input. What's cool is that this gets much faster than you'd think, even while in the middle of "full" areas of the disk. On any drive that isn't full, you'll have huge chunks of 0s, further speeding up DD. So far, at least, DD is just about running along at my hard disk's theoretical transfer speed: ~116.4 MB/second, and it hasn't yet reached those large blank areas.

Give these options a try – they work! Please note: CAREFULLY change if= and of= to properly point to the correct drives listed by (for Macs):

diskutil list

Answer 5

For the record, in macOS High Sierra at least, /dev/disk appears to be much faster than /dev/rdisk. Running either dd or ddrescue, my throughput comparison copying from a magnetic HD to an SSD was 3.7MBps using /dev/rdisk, and 45MBps using /dev/disk. So, on later versions of macOS, it may be best to use /dev/disk instead of /dev/rdisk for best performance.

Answer 6

I think before argue which path node is faster or dive into a serial tests. We should consider other factor that will dramatically affect final read/write speed.

like Micro SD card spec, class 4/10/HC I ... sd card reader chip and interface, usb 1.1/2.0/3.0/3.1 os total memory/free memory, os load, os harddisk type, HDD/SSD, HDD spin speed and cache size, SSD size/cache/free space/ os harddisk interface, ata/sata/esata,

if any factor became a bottleneck, then we will got false conclusion.

here is my result: osx 10.12.6, ssd,

read microSD 16G by a usb 2.0 card read and write to external 3.5 inch HDD by usb 3.0,

15193+1 records in
15193+1 records out
15931539456 bytes transferred in 1423.067033 secs (11195214 bytes/sec)

write microSD 32G thru internal card read and data source is external 3.5 inch HDD by usb 3.0,

0+253945 records in
0+253945 records out
15931539456 bytes transferred in 440.093686 secs (36200336 bytes/sec)

you can see, write speed > read speed !!

Answer 7

Many of the speed discrepancies are quite likely the result of the type of storage media being used, as a consequence of the mechanism of read or write access.

It may be faster to use /dev/disk with storage media that physically spins, like HDD & tape drive (magnetic) or CD & DVD (optical). The buffered stream would allow time to properly position the disk and to either make the physical change to the disk (via magnetic head or laser) or allow the drive hardware to pick up the data signals from the disk.

It's probably faster to use /dev/rdisk with storage media that is solid state & electronically controlled, like SSD & SD card or USB storage drive. There is no need to physically reposition anything to access any other data location so seek time is almost instantaneous, and the read or write process only involves electricity within a circuit instead of requiring high-resolution magnetic field fluctuations or precision-timed laser beams which have to travel a small distance to reach the storage medium. Any time used to fill the software buffer would be spent "holding" the information until it is deemed useful, but with no processing bottleneck ahead of it, it becomes useful as soon as the hardware bus can take it (it can usually take it immediately, unless it's already being fed at its maximum throughout rate).

Disclaimer: This is based on logic, not experimentation, and my understanding of computer systems and hardware, which is not yet complete in all aspects.