Without knowledge, I would normally expect SATA, SCSI and USB Mass Storage to be driven by different drivers. In Linux however, the SCSI subsystem drives all of them.
Why does Linux use the SCSI subsystem to drive even things not clearly related to SCSI? Why aren't NVMe and (historically) PATA support part of the SCSI subsystem?
Without knowledge, I would normally expect SATA, SCSI and USB Mass Storage to be driven by different drivers. In Linux however, the SCSI subsystem drives all of them.
Well, USB Mass Storage is literally SCSI under the hood; it uses the actual SCSI command set, merely over another transport. (Even more so if your disk supports UAS, which stands for "USB-Attached SCSI", but it applies just as much to the basic BOT storage devices.) Which means, yes, your external USB-SATA disk enclosures are translating the commands between ATA and SCSI (with a special 'ATA passthrough' command for things like SMART access).
(And if you ever had one of those FireWire HDDs or iPods back in the day, those were SBP-2 devices, which is again SCSI. It's a pretty versatile command set.)
SATA support goes through SCSI because at the time SATA was introduced, the existing Linux PATA subsystem (which dated to the earliest days of Linux) was kind of garbage in comparison to the robust SCSI subsystem, and SATA was already somewhat closer to SCSI in terms of capabilities – queueing, tagged commands, etc. – so treating AHCI as a weird kind of SCSI controller (with a SCSI-ATA translation layer, "libata") likely offered much better performance than treating it as a weird kind of PATA controller.
(Indeed even many Windows systems at the time had vendor SATA drivers that presented the SATA AHCI controller as if it were an SCSI controller, before Windows gained native AHCI support – e.g. Windows XP knew SCSI but not AHCI.)
This support uses the official SCSI-to-ATA translation document produced by the T10 working group responsible for SCSI, and apparently most if not all things have a 1:1 translation so I think it's not nearly as much of a hack as it would seem. Still, it seems that at the time this was thought to be a temporary solution, with plans to make a new Linux subsystem for SATA devices, but that never happened; I guess 'libata' (the translation layer) turned out to work better than expected.
(And as a last note, all ATA CD/DVD drives use the "ATAPI" protocol – which is just SCSI inside ATA packets. Again, even on Windows, you'll see nearly all CD/DVD-burning software list CD/DVD drives as if they were SCSI devices, because in a sense they are.)
Why aren't NVMe and PATA support part of the SCSI subsystem?
PATA support is part of that. For many years now, even PATA disks have been showing up as /dev/sd* devices via the 'libata' SCSI-ATA translation rather than actual PATA /dev/hd* devices. The "legacy" PATA drivers were completely removed from Linux in recent months, but they haven't been in use for a long time.
As for NVMe, I assume it is the same as what was mentioned in the T10 mailing list message regarding SATA, where a separate subsystem was the eventual goal in the first place... and given that NVMe was designed as a successor to both ATA and SCSI several decades later, there might have been just too many differences for it to be practical, e.g. the extremely deep queues that NVMe allows compared to what SCSI was designed for.
That is to say, libata worked well because PATA/SATA was a subset of SCSI capabilities, whereas NVMe is more of a superset.
(The NVMe subsystem originally did have a similar kind of SCSI translation but that eventually was removed; the commit message claims it was in part due to significant differences between the two.)
This is explained in the SCSI Interfaces Guide :
Although the old parallel (fast/wide/ultra) SCSI bus has largely fallen out of use, the SCSI command set is more widely used than ever to communicate with devices over a number of different busses.
SCSI commands can be transported over just about any kind of bus, and are the default protocol for storage devices attached to USB, SATA, SAS, Fibre Channel, FireWire, and ATAPI devices. SCSI packets are also commonly exchanged over Infiniband, I20, TCP/IP (iSCSI), even Parallel ports.
The explanation is that the SCSI subsystem is used for its mastery of the versatile SCSI protocol across all sort of busses, so not principally to control SCSI devices.
More details for the SCSI interface can be found in the above article. The general system structure is described as:
The SCSI subsystem uses a three layer design, with upper, mid, and low layers. Every operation involving the SCSI subsystem (such as reading a sector from a disk) uses one driver at each of the 3 levels: one upper layer driver, one lower layer driver, and the SCSI midlayer.
The SCSI upper layer provides the interface between userspace and the kernel, in the form of block and char device nodes for I/O and ioctl(). The SCSI lower layer contains drivers for specific hardware devices.
In between is the SCSI mid-layer, analogous to a network routing layer such as the IPv4 stack. The SCSI mid-layer routes a packet based data protocol between the upper layer’s /dev nodes and the corresponding devices in the lower layer. It manages command queues, provides error handling and power management functions, and responds to ioctl() requests.
It's the lower layer drivers that make it possible for the SCSI subsystem to interface with so many types of busses and devices.
PATA is now outdated and SATA is the standard for most consumer-grade storage devices. NVMe is used for performance, particularly for SSDs. SCSI is still used for specialized enterprise environments for compatibility and its advanced features.
"SCSI" is an umbrella term for a system of related, layered protocols and definitions. SCSI specifies everything from the physical dimensions of the connectors and cables, the electrical voltages and currents on the wire, the clocking mechanism, the protocol on the wire, to the commands that devices understand.
Obviously, the connectors, cables, electrical specifications, and wire protocols are obsolete nowadays, and have been for a long time.
The commands, however, live on. An Operating System doesn't care about the size of the plug or the voltage on the wire or how data is encoded on the wire, it only cares about the commands it sends to the devices. So, as long as a device "understands" the SCSI commands, from the OS's perspective, it is a SCSI device. The OS doesn't know how the device is connected to the system, it only knows the "language" the device speaks.
SCSI has always been the "more professional", "more powerful", "more mature" alternative compared to its competitors (most notably ATA), so it makes sense that lots of newer technologies simply adopt the SCSI commands instead of going through the trouble of creating their own.
The first CD drives which were introduced were insanely expensive and intended for Unix workstations, so naturally, they were SCSI drives. When the first ATA CD drives came out, instead of re-inventing everything the SCSI guys had done, the ATA guys invented ATAPI, the ATA Packet Interface, which is nothing but a way of tunneling SCSI commands through ATA.
When ATA hard disks gained more "professional" features such as self-monitoring, encryption, command queueing, the ATA guys did not go off into the woods and did their own thing, they took heavy inspiration from the SCSI features.
When Firewire was introduced, they chose to use SCSI commands for external drives. When USB was introduced, they did the same.
Interestingly, there has also been technology transfer in the other direction: the SAS protocol and the electrical and physical wiring is heavily based on SATA.
When CD-ROMs were introduced into Linux, they were based on the SCSI subsystem, which led to the interesting state that a hard disk and a CD-ROM connected to the very same cable would show up in two different subsystems.
When USB and Firewire were introduced, naturally, they used the SCSI subsystem.
So, when SATA was introduced into the Linux kernel, it was decided to base it on the much cleaner and much better implemented SCSI subsystem rather than the old and crufty ATA subsystem.
And at some point, it was decided to move PATA hard disks to the SCSI subsystem as well.
Why does the Linux SCSI subsystem drive hardware not obviously related to SCSI?
To steal the comment under the question: because they are related to SCSI, just in a non-obvious way.
They all use the SCSI commands, which from the perspective of the OS, is the only thing that's relevant.
Why aren't NVMe and (historically) PATA support part of the SCSI subsystem?
NVMe is specifically designed to be unlike traditional storage command APIs because flash memory is very unlike traditional rotating hard disks or tapes.
For PATA, it is historical. I wouldn't be surprised if the real reason is simply that Linus's computer in 1991 didn't have a SCSI disk, so the PATA subsystem came first. After SATA support was introduced, PATA was indeed moved to the SCSI subsystem.
