6

During the last years I have accumulated a quite heterogeneous set of hard disks of different sizes and speeds for storing my private data. I'm planning to put them in a self build Linux filer to reduce the hassle of replicating local data manually, reducing duplicated files and better utilize the given resources. Also I'm expecting that in the coming years my storage need will increase. Therefore it should be possible to dynamically add disks and also removing single drives to replace them with newer and larger ones.

If I'm not mistaken the most common option for building a filer would be using a (software) RAID to increase reliability and an additional backup scheme on external or removable drives to prevent accidental loss of important data.

As RAID5/RAID6 needs drives of equal size it doesn't satisfy my need for a more dynamic disk addition/removal scheme. So I'm looking for a FLOSS file system or block device abstraction layer that provides:

  • dynamical addition and removal of hard disks possible
  • replication/redundancy similar to RAID5, maybe adjustable per file or directory
  • no additional (especially dedicated) machines necessary (but the possibility to add a second machine later if needed would be nice but is not necessary)

I looked a bit in distributed file systems like XtremeFS, but haven't found one yet that satisfies all points and work well on a single machine. Do you have an idea what could be a solution?

Randolf Richardson
  • 14,794
Moredread
  • 75

4 Answers4

3

To put it simply, believe it or not, hardware solution are easier to rebuilt and more reliable than software solution, and existing solution are often less costly to implement.

Building your storage system is generally not a good idea unless you are very well versed in the art of storage management. In general, the old dictum "If you had to ask it's not for you" applies.

Bottom line: I would recommend buying new harddisks (These days those are cheap compared with my data - if your data isn't worth buying a set of harddisk these days, Houston we've had a problem.) and setting up a brand new raid5/6 to handle it, together with some cold replacements.

More importantly, note that backup and redundancy is two separate entity in storage management. Redundancy is for maintaining the service ONLINE in case of outage of one single piece of equipment, and backup is in case the whole thing fail. For example, if the power supply suddenly fails and try to pump some 100V AC onto the 12V rail (Not that it is likely, but...) nevermind RAID5, nevermind RAID6 - all would be gone. You will need backups to handle those situation. For backup, follow these simple rules:

  1. Take backups offline
  2. Take backups offsite
  3. Do backups often

Good luck!

bubu
  • 10,029
2

I'd look into ZFS. It (or some implementations of it) offers some of the things you specifically ask for (redundancy across heterogenous media, ability to readily cope with loss of a spindle (if redundant data exists)).

BTRFS is theoretically heading in that direction, and I don't know that ZFS has the features mentioned above under Linux (or even Solaris) at this point, but at least it is an avenue of investigation.

Beyond that, I'd look into LVM and creative partitioning + MD or LVM features for RAID-like traits. LVM should let you resize partitions (they call it something else) without too much difficulty. I wouldn't consider this a low-maintenance approach though.

Slartibartfast
  • 8,368
1

On linux use AUFS to portray the disks as a single filesystem and then SnapRAID for protection. There's a straight forward write up of How I configured Snapraid and AUFS by Wouter Dullaert.

There are alternatives to AUFS such as Btrfs and OverlayFS. OverlayFS has been merged into the Linux kernel so long term that may be a more supported route than AUFS.

Windows allows mounting partitions as directories from the Disk Management tool, and from there use SnapRAID as above (it's multiplatform)

matt wilkie
  • 5,324
0

It should be possible (and I'm sorry, I'm not very deep into LINUX) to build multiple RAID volumes across a heterogenious set of disk drives; thus maximizing the use of space while keeping redundancy.

Imagine you have 3x1TB drives, 2X2TB, and 1 4TB drive. You can build a RAID 5 volume of 5TB of space using 1TB from all 6 drives. You can build another RAID 5 volume of 2TB using 1TB from each of the remaining 3 drives.

In Windows small-business server, these volumes are then spanned; appearing as a single volume. I wish I could be more helpful in making a specific recommendation regarding how to do that in LINUX.

Should you add, say, another 4TB drive: you could then grow both arrays by 1TB and add a thrid 2TB array using the 2TB space available on each of the 4TB drives.

As to hardware-vs-software: Hardware is undoubtably faster; but costs more and, should the hardware controller itself fail, may require you find an identical controller or lose all data.

As always: redundancy is not a replacement for backups.

JerryLove
  • 1