Linux software RAID robustness for RAID1 vs other RAID levels

Question

I have a RAID5 array running and now also a raid1 that I set up yesterday. Since RAID5 calculates parity it should be able to catch silent data corruption on one disk. However for RAID1 the disks are just mirrors. The more I think about it I figure that RAID1 is actually quite risky. Sure it will save me from a disk failure but it might not be as good when it comes to protecting the data on disk (who is actually more important for me).

1. How does Linux software RAID actually store RAID1 type data on disk?
2. How does it know what spindle is giving corrupt data (if the disk(subsystem) is not reporting any errors)

If RAID1 really is not giving me data protection but rather disk protection is there some tricks I can do with mdadm to create a two disk "RAID5 like" setup? E.g. loose capacity but still keep redundancy also for data?

Answer 1

Since raid5 calculates parity it should be able to catch silent data corruption on one disk.

Nope. One could totally obliterate a sector with random data, and a RAID5 would not bat an eyelash. RAID1 has the same problem.

In general, RAID does not provide real-time data integrity checking. What it does provide is fault-tolerance in the face of one (or more, with some RAID levels) drive failures. Those are two very different things.

If you are looking for something to catch file corruption, you need filesystem support. RAID does not do it. At least, not on its own.

To answer your specific questions:

RAID1 is implemented simply as two (or more) identical mirrors. When the mirrors do not agree on the contents of a sector, then corruption has occurred. The thing is, the RAID system is not often in a position to be aware of this, since it does not normally read all the mirrors when it is asked to retrieve a given sector. For efficiency, it will likely just schedule one disk to read it (hopefully the one whose heads are currently nearest to it).

Suppose that, during a "scrubbing" operation, when the RAID system is explicitly asked to verify the consistency of all its data, an inconsistency is discovered. The question of how to resolve this inconsistency has no simple answer. Note that this problem equally affects RAID5 as it does RAID1, and other RAID levels as well.

In a RAID1, an inconsistency appears as two mirror sectors containing different data. How does the RAID system decide which sector represents the correct data? Well, that is an implementation detail, and I honestly don't know how the Linux system is implemented exactly. But the problem is fundamental: the mirrored sectors are different, and there may be no indication as to why they have become that way. So the best the RAID system can do is to flip a coin: choose one at random to be the "correct" data.

In a 3-disk RAID5, an inconsistency appears in the form of a triple of sectors whose parity sector is incorrect. The question is: which of the 3 sectors is wrong? Again, there is no obvious answer. Any of the three could be corrupt, and there's probably no way to know. If you must choose one sector to be recalculated from the other 2, you have 1 in 3 chance of choosing the one that was actually corrupt. This demonstrates that RAID1 is actually "safer" than RAID5, in this sense. The RAID1 has a 50% of choosing the wrong sector, while the RAID5 has a 67% chance of choosing wrongly.

To summarize: RAID is not designed to catch disk errors as they happen. RAID provides fault-tolerance in the face of whole-drive failures. Nothing more.

Answer 2

RAID5 will not catch silent data corruption on disk; you need a filesystem such as ZFS or BTRFS, with block-level checksumming, to protect against that. RAID5 also will not perform as well as RAID1 due to its parity calculations. With any type of parity RAID, you should take care to address the RAID5 write hole, which introduces the potential for data corruption in the event of a power outage, for example.

Linux software RAID1 has an interesting advantage in that you can create as many mirrors as you want--so if uptime is your #1 priority, you can configure an 8-disk RAID1 that maintains 7 redundant copies.

The potential for data corruption using RAID1 is about on par with using a single disk (with no RAID).

If you're really concerned about data corruption, you should either use a checksumming filesystem or regularly compare your unchanged data with several backups. A popular ZFS success story tells the tale of a person whose computer was silently corrupting his data, and he didn't even know it until he started using ZFS. After a little troubleshooting, he figured out that the cause was a faulty power supply.

You should also consider that the hard drive is not the only place where the data could be corrupted. If you're not using ECC RAM (and an enterprise-grade motherboard that is not only compatible, but enables ECC), for example, a cosmic ray could flip a bit in-memory. Depending on what kind of data we're talking about, it may not even matter. If it's a video or music file, the bit flip will go unnoticed when you play back the file.

When you get down to the meat of the issue, silent data corruption is all about probabilities. The probability that your data will be corrupted is not very high; otherwise we'd all be cursing constantly at our data being corrupted yet again. (Everybody would probably keep multiple backups and even hard-copies of everything, because they wouldn't trust the computer to keep one good copy.) The likelihood that you'll even notice the data corruption is even lower. Most people are completely unaware of the concept of silent data corruption, and they get by just fine. It's also worth noting that even many enterprise-grade disk storage systems don't protect against silent data corruption at the filesystem level. But if you're not a gambling person at all, you might want to throw some money at enterprise-grade hardware (ECC RAM, battery-backed disk controllers, and all), and switch to using ZFS or BTRFS.

Answer 3

Focusing on the actual questions...

Even RAID 5 will not be able to correct silent bit rot, but it can detect it during a data scrub. Though it will be able to correct a single block that has been reported by the disk as having an Unrecoverable Read Error (URE). Note that not all drives in a RAID5 stripe are read from for a normal data read, so if the error exists in the stripe on the unused disk it will go undetected until you perform a data scrub. Silent bit rot detection with any standard RAID can only occur during data scrubbing. RAID 5 cannot do even this during a rebuild of a failed disk, this is what most concerns these days are with RAID 5.

1. Linux mdadm RAID 1, like nearly all RAID 1 implementations, is just duplicating/mirroring the same data on to multiple disks. It adds no error correction or detection data. If you take a disk out of any RAID 1 and use it in another PC, it will very likely just work as a normal single disk. Linux mdadm adds some array description to the start of the disk so it can know which partitions belong to what array, so mdadm will know it was a RAID 1 but can mount and use the single disk anyway.
2. All RAID 1 controllers, be they software or hardware, rely on the fact that HDDs use their own error detection and correction methods. See this wikipeadia article for some info on how HDDs do this, in particular note the use of Error Correction Coding (ECC).

This is why most bit rot will be reported as an Uncorrectable Read Error (URE) by the disk systems to mdadm. However there are still risks to your data that will not result in any error being reported by the disk such as

• if there was a head positioning error during a write so some random nearby sector is overwritten with data and correct ECC data for that block. Reading the block that was actually written will report that it read the block just fine, even though it is not.
• the server lost power before it had written its data to all of the disks in the array, then some blocks in that stripe will be in disagreement with the others.

and other types of errors such as those described on the ServerFault page Is bit rot on hard drives a real problem? What can be done about it?

RAID 6 and RAID 1 arrays with at least 3 disks are the only standard RAID levels that have the potential to be able to detect and correct some forms of silent bit rot that are not reported by the individual disks as errors, by using a forward error correction style voting system. Though I do not know if mdadm implements the required code for this mdadm does NOT do this, mdadm ONLY works if the disc reports the error.

If a raid controller did implement that voting style system bit rot could only be corrected:

• For RAID 6 - only if the error is in one of the parity blocks. This is due to the possibility of a 3 way vote between data, parity 1, and parity 2. If parity blocks 1 or 2 say there is an error but the other 2 do not, then the parity block can essentially be out voted. The reason it cannot correct the problem if the error is in one of the data blocks is that it cannot know which data block has the error, unless it is a 3 disk raid 6, which are typically not allowed. I doubt that any implementation, would bother with such an obscure correction scheme and just report it as an error.
• For RAID 1 with 3 or more active supposedly already synchronised disks it can conduct a simple majority vote. Though again, I don't know if any RAID implementation bothers with this logic as not many people use a 3+ disk raid 1. If it did implement the required logic a RAID 1 that
• normally had 3 disks, a block with silent bit rot could be auto-corrected, though not if it was during a rebuild as that would reduce the number of active sync'd disks to 2.
• a 4 disk raid 1 could auto-correct any stripe with a single bad block even during a rebuild of 1 failed disk.
• a 5 disk can auto-correct a stripe with 2 silently bad blocks, though that is reduced to 1 if it is found during a rebuild of 1 or 2 simultaneously failed disks.

FYI I noticed that the Synology DS1813+ devices use mdadm for both data and system partitions and it uses RAID 1 across all 8 disks for the system partitions.

As you may have observed this places a lot of reliance on the disk being able to report bad data as an error. While everyone is saying to use ZFS to solve this issue. I believe no longer believe, ZFS's main data integrity improvements are that it provides more frequent data scrubbing due to it checking mirrors/parity with every read, and independent block level parity (which means many silently corrupted blocks are no longer silent and corrected if possible) and it may implement the above logic for silent data corruption.

To test if a particular system can detect and/or correct silent data corruption use the Linux dd command to write random data to one of the partitions in the array, then test if the data is still good on the array. Warning do not do this test on a system with data you want to keep as your system may fail the test. For standard RAID levels you will need to perform a data scrub between corruption and test read.

Edit: I've since tested mdadm vs ZFS on an array of cheap USB sticks which do not have much, if any, built in error detection capabilities. The 3 stick mdadm RAID1 array failed badly with much data loss, even with twice a day data scrubs. While the ZFS array using the same layout worked flawlessly, even when I purposfully used known bad usb sticks from the previous test with mdadm. This highlights that the benefits of ZFS extend well past frequently checking the data when compared to mdadm.

Answer 4

Neither raid1 nor raid5 will protect you from silent bit corruption if the corruption is done by the harddisk.

Think about it: With both raid1 and raid5 it is easy to detect that data have been corrupted, but there is no way to determine which disk got the good data, and which got the bad.

But remember: Silent bit corruption on a harddisk happens REALLY REALLY seldom(The harddisk got its own internal checksum on all blocks), and raid does not remove the need for backup.

If you need to survive silent disk corruption, use something like raid6, or a filesystem which checksum all its files.