How to find which drives are mirrored within a RAID-10 array?

Question

I have a home media server running on openSUSE 12.2. I'm using eight 2TB drives in a RAID-10 configuration. I deliberately bought two different types of drives: four Seagate Barracuda Green and four Western Digital Red. My goal is to configure the RAID such that each mirrored pair within the array will consist of dissimilar drives (ie, one Seagate drive and one WD drive). YaST2 Partitioner unfortunately did not give me this level of choice in the structure of the array, so now I'm trying to find out what the default RAID10 structure looks like.

I do know the following:

sdc, sdd, sde, and sdf are all WD drives
sdg, sdh, sdi, and sdj are all Seagate drives

I chose the default 'n2' layout when creating the RAID. My guess based upon info from these two sources is that adjacent drives are mirrored (ie, sdc==sdd, sde==sdf, etc…), but I want to know for sure:

Here is the output of mdadm --detail /dev/md0:

/dev/md0:
        Version : 1.0
  Creation Time : Sat Mar 16 12:55:11 2013
     Raid Level : raid10
     Array Size : 7814045696 (7452.05 GiB 8001.58 GB)
  Used Dev Size : 1953511424 (1863.01 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sat Mar 16 13:09:37 2013
      State : active, resyncing

Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
     Layout : near=2
 Chunk Size : 2048K


Resync Status : 1% complete
       Name : aldaris:0  (local to host aldaris)
       UUID : c6cc3943:97394500:b77d44cd:f02ed876
     Events : 149

Number   Major   Minor   RaidDevice State
   0       8       33        0      active sync   /dev/sdc1
   1       8       49        1      active sync   /dev/sdd1
   2       8       65        2      active sync   /dev/sde1
   3       8       81        3      active sync   /dev/sdf1
   4       8       97        4      active sync   /dev/sdg1
   5       8      113        5      active sync   /dev/sdh1
   6       8      129        6      active sync   /dev/sdi1
   7       8      145        7      active sync   /dev/sdj1

And here are the contents of /proc/mdstat:

Personalities : [raid10] md0 : active raid10 sdj1[7] sdi1[6] sdh1[5] sdg1[4] sdf1[3] sde1[2] sdd1[1] sdc1[0]
      7814045696 blocks super 1.0 2048K chunks 2 near-copies [8/8] [UUUUUUUU]
      [>....................]  resync =  4.8% (375163456/7814045696) finish=1206.5min speed=102751K/sec
      bitmap: 57/59 pages [228KB], 65536KB chunk
unused devices: <none>

So my questions are:

How do I tell which drives are mirrors of each other?
Is there a way to change this, or should I just swap the wires around (since that will swap the drive letters) and then rebuild the RAID?

Tangential note, for anyone wants to know my reasoning for doing this is:

Drives of the same model and batch, operated under similar usage loads, uptime, and temperature have little systematic variation, and differences in time to failure between drives will be primarily driven by random variation in the manufacturing process. This increases the risk of multiple drives dying at once. By purchasing drives not just from different batches but completely different manufacturers, I am introducing systematic variation into my array, thus influencing which drives will fail at similar times.

score 10 · Answer 1 · answered Aug 26 '16 at 08:37

Recent versions of mdadm show this right in the details of the array. Example from mdadm v3.3 - 3rd September 2013

 $ mdadm --detail /dev/md1

/dev/md1:
        Version : 1.1
  Creation Time : Tue Aug 23 11:45:41 2016
     Raid Level : raid10
     Array Size : 3864803328 (3685.76 GiB 3957.56 GB)
  Used Dev Size : 1932401664 (1842.88 GiB 1978.78 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Fri Aug 26 09:39:28 2016
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : px4-300r-THXOAP:1  (local to host px4-300r-THXOAP)
           UUID : 5ee06437:83dfdb64:808feaa2:5d57b1e6
         Events : 620

    Number   Major   Minor   RaidDevice State
       4       8       50        0      active sync set-A   /dev/sdd2
       1       8       34        1      active sync set-B   /dev/sdc2
       2       8       18        2      active sync set-A   /dev/sdb2
       3       8        2        3      active sync set-B   /dev/sda2

Note the denotation set-A or set-B. In the above case, sdd and sdb can fail together without data loss. It is possible this data is not available while the array is rebuilding though.

score 3 · Answer 2 · edited Aug 16 '24 at 17:25

I had the same issue and after googling a while I didn't find a reliable answer. After giving it some thoughts, I figured that the mirrors have the same data and so we could compare some part of it.

Note: Be careful, if you have more than 2 drives with the same checksum you are probably comparing empty diskspace, choose another offset (skip option).

With this few commands, you can figure it out:

for disk in sda sdb sdc sdd
do
  echo -n "$disk = ";
  dd if=/dev/$disk skip=1M bs=1M count=1 2>/dev/null | md5sum;
done

This will output something like:

sda = 7c4ef0f3e0143b35e044d5d65908a3a2  -
sdb = 7c4ef0f3e0143b35e044d5d65908a3a2  -
sdc = e02f7d61ad3791bd691da5b7516928a5  -
sdd = e02f7d61ad3791bd691da5b7516928a5  -

Now we know that sda/sdb is one mirror and sdc/sdd another one. One of each must stay to avoid data loss.

The "dd" command is reading one time (count=1) one Megabyte (bs=1M) at one Megabyte offset from the disk start (skip=1M). Don't skip=0, because the begining of the disk contains different information. The data usually begins after 1MB.

score 2 · Answer 3 · answered Sep 04 '14 at 14:22

You can always verify which are mirrors by a crude manner of comparing the data, for example:

# dd if=/dev/sda1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
7c01afa434fc74aeddc8ec0546e4c332  -
# dd if=/dev/sdb1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
1f0f8166857710d555cb35db4a23891a  -
# dd if=/dev/sdg1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
7c01afa434fc74aeddc8ec0546e4c332  -
# dd if=/dev/sdf1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
1f0f8166857710d555cb35db4a23891a  -

(if you don't get any matches, you may need to increase skip=, as you're not skipping over RAID superblocksl; and if you get same md5sum for more than 2 disks, you may need to increase count=, as you're probably reading and m5summing zeros - to prevent that you should put some data on the raid first, otherwise they might be full of zeroes)

As for the swapping wires around, you don't need to do that - mdadm should create raid with devices as specified on command line in mdadm --create, so you would just specify drives in different order on command line.

Nuitari · Answer 4 · 2024-08-16T20:32:09.107

I know this is a very old question, but this is also the top result in google for that problem.

This was tested with an Ubuntu 22.04 VM. MDADM version 4.2 Kernel version 5.19.0-32-generic

I created a 10 device RAID 10 array, mkfs, then added about 80% of random test data and check summed it.

Using the dd technique above I saw how the mirrors were laid out.

# for i in /dev/vd?3; do echo -n "$i = "; dd if=$i skip=1 bs=1M count=10 2>/dev/null | md5sum; done
/dev/vda3 = efd93e089fe3ccd55c7430f0ed8dd555  -
/dev/vdb3 = efd93e089fe3ccd55c7430f0ed8dd555  -
/dev/vdc3 = 620d252f6429da8681637e6a3714d039  -
/dev/vdd3 = 620d252f6429da8681637e6a3714d039  -
/dev/vde3 = 25dd1593fac22714fad11c32b1a87a12  -
/dev/vdf3 = 25dd1593fac22714fad11c32b1a87a12  -
/dev/vdg3 = e3dc8c0cb8032bfabc15644f3908e003  -
/dev/vdh3 = e3dc8c0cb8032bfabc15644f3908e003  -
/dev/vdi3 = a15424d37488bf6c59e4fa355cd8674d  -
/dev/vdj3 = a15424d37488bf6c59e4fa355cd8674d  -

I remounted the filesystem with the sync flag. This avoids a lot of issue with caching.

Then I decided to fail 2 drives that had the same checksum to see what happens.

# mdadm /dev/md1 --fail /dev/vdg3 /dev/vdh3 
mdadm: set /dev/vdg3 faulty in /dev/md1
mdadm: set device faulty failed for /dev/vdh3:  Device or resource busy
cat /proc/mdstat
Personalities : [raid1] [raid10] 
md1 : active raid10 vdj3[9] vdi3[8] vdh3[7] vdg36 vdf3[5] vde3[4] vdd3[3] vdc3[2] vdb3[1] vda3[0]
      78597120 blocks super 1.2 512K chunks 2 near-copies [10/9] [UUUUUU_UUU]

Ok, so it didn't let me set 2 mirrored drives to faulty, this is a nice safeguard.

Then I set faulty all drives in the array one by one. Everytime I got the error

mdadm: set device faulty failed for /dev/vde3:  Device or resource busy

However, the array now looks like this:

# cat /proc/mdstat 
Personalities : [raid1] [raid10] 
md1 : active raid10 vdj3[9] vdi3[8](F) vdh3[7] vdg3[6](F) vdf3[5] vde3[4](F) vdd3[3] vdc3[2](F) vdb3[1] vda3[0](F)
      78597120 blocks super 1.2 512K chunks 2 near-copies [10/5] [_U_U_U_U_U]

Removing the failed drives work:

# mdadm --remove /dev/md1 failed
mdadm: hot removed 252:3 from /dev/md1
mdadm: hot removed 252:35 from /dev/md1
mdadm: hot removed 252:67 from /dev/md1
mdadm: hot removed 252:99 from /dev/md1
mdadm: hot removed 252:131 from /dev/md1

The underlying reason I'm doing this is because 2 of the devices are much bigger than the rest and I need to restructure everything to make use of that space.

Of course this doesn't protect you at all if another drive fails as at that point you have the functional equivalent of a raid0.

Checking that the data is usable:

# sysctl -w vm.drop_caches=3
# md5sum garbage garbage2; cat *sum
c5a68e3bfcce576ed09c572134c8d7b7  garbage
741b2c386fdecc47f8e26e7b5bc481fe  garbage2
741b2c386fdecc47f8e26e7b5bc481fe  garbage2
c5a68e3bfcce576ed09c572134c8d7b7  garbage

All good! Of course, make sure you have backups, a drive can absolutely fail while you do this.

In one of my tests, without the sync flag, failing a lot of drives quickly did result in filesystem corruption.

score 0 · Answer 5 · answered Apr 16 '13 at 19:52

I think you are talking about a real raid 10 array (1+0 striped mirroring)

sdc/sdg = md0 raid1 2TB |
sdd/sdh = md1 raid1 2TB |_  md4 raid0 8TB
sde/sdi = md2 raid1 2TB |
sdf/sdj = md3 raid1 2TB |

1. Create your 4 raid1 arrays:

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/{sdc,sdg}
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/{sdd,sdh}
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/{sde,sdi}
mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/{sdf,sdj}

At this time you have 4 raid 1 arrays with 2TB space on each,

2. Let's assemble them!

mdadm --create /dev/md4 --run --level=0 --raid-devices=4 /dev/md{0,1,2,3}

--run options is usefull because some components are active in another array

3. Adapt your mdadm.conf file

You may need (depending of your configuration) to adapt this file to reflect changes about our new array (/dev/md4).

4. Enjoy.. Your new DIY raid10 array!

score -1 · Answer 6 · answered Sep 05 '14 at 12:48

-1

Run "mdadm --examine device" on each component device (i.e. /dev/sda1, /dev/sdb1, etc.). The information there should help you determine which components are each others' mirror.

answered Sep 05 '14 at 12:48

wurtel

1,575

How to find which drives are mirrored within a RAID-10 array?

6 Answers6

cat /proc/mdstat