1

I'm currently running personal NAS storage.
I'm not professional in computers and
I'm just using copy-paste to do backup. (no raid or something like that)

I have 8TB * 8 seagate archive drive.
4 drive for nas, and 4 drive for backup.
I rarely update/write on this drives.

I experienced some errors on the video file.
So I had to re-write the files.

But when I checked the bad sector or other errors by using Crystaldisk
I couldn't find any errors..

I know nothing how hdd drive works
here is the question

  1. Is it possible to check little error/corruption on hdd drives? Like a one single bits corruption..

  2. Best way to manage this kind of archive hdd drives?
    I was thinking of making an md5 hash for all files.
    But there should be an better / smarter way to do this.

All data is very important for me, I'm willing to pay many money for this..
How does the cloud storage services manage their data safely?

Attie
  • 20,734
J. Dumb-ass
  • 11
  • 3

1 Answers1

5

Many commonly used filesystems such as FAT, NTFS, ext2/3/4 do not ensure the integrity of the file data (for example with checksums)... this coupled with the real threat of bit-rot can lead to the situation you find yourself in... Data is returned from the hard disk without error, but it isn't correct.

There is a useful wiki page comparing filesystem features here - for your use-case, we are interested in the "Data Checksums" column of the "File Capabilities" table, here.

At the time of writing, the list of filesystems that checksum the data (and that you may plausibly have access to) is: ZFS, Brtfs, ReFS.

Using a more modern filesystem such as these allows better resiliency against bit-rot. I can't comment on the others, but use ZFS extensively myself, so that's where the rest of my answer will focus (specifically on the assurance of data, ignoring many other useful features).

Single Disk / Stripe

Using ZFS with a single disk or striped pool (many disks with no dedicated parity disk) provides you with the peace of mind that if data is returned, it is correct. If the data cannot be read correctly, then you will see an error. This is achieved by storing a checksum of the data as well as the data itself.

Note, that even with such a pool, the ZFS copies property may help with simply assuring data integrity - however it provides no benefit with regard to device failure, so I would advise against its use.

copies=1 | 2 | 3
    Controls the number of copies of data stored for this dataset. These copies are in
    addition to any redundancy provided by the pool, for example, mirroring or RAID-Z.
    The copies are stored on different disks, if possible. The space used by multiple
    copies is charged to the associated file and dataset, changing the used property
    and counting against quotas and reservations.

Mirror or RAIDZ

Using ZFS with a mirror, or raidz1 or raidz2 pool (1 or 2 parity disks) enhances the above ability to detect errors, and allows the filesystem to attempt to automatically heal the damaged data. If a read fails (due to mismatched checksum), then a read request will be sent to one of the mirror or parity disks. In the very unlikely event that all sources have bad checksums you will see an error, as above. But in the typical case the mirror or parity disk will read successfully, the correct data will be returned to you, and the damaged data will be written afresh.

Unfortunately I'm not currently aware of a consumer NAS product that provides ZFS or Btrfs support, but if you are running this on a PC, then distributions like FreeNAS (uses ZFS) may be very interesting to you.

You could of course calculate a checksum for all files as you've mentioned, but this leads to problems when attempting to maintain and verify the checksums.

Attie
  • 20,734