5

Situation

I have huge amount (around 1 TB) of mixed data (photos, documents, pdfs and other kind of file formats). All data is arranged in many folders. As my laptop drive is keep filling , I have to move this data to some kind of long term storage. I have decided to transfer this data to one regular external hard drive as well as one NAS device.

Question

Is it good idea to zip everything and transfer to these devices ? I am planning to use 7Zip for this process. It might save some space also can increase transfer rate. Further can also save any unexpected deletion of files by any antivirus.

Further Problem

Last time I was transferring such huge data, there were some files which I could not transfer due to some error or corruption. If something like this happens for zipped archive, then I am screwed.

Suggestion about any other better way to storing and transferring such big data is also welcome.

Dexter
  • 151

2 Answers2

4

A few points come to mind:-

  • Space saving: many files, especially multimedia, are already in a compressed format, and little space will be gained by trying to compress further (indeed, I have seen multimedia files in a ZIP archive that are larger than the original, though the ZIP program should recognise that this will happen and store without trying to apply further compression).
  • Access times: a large archive can take a long time to access and update.
  • Access security: using a ZIP password gives cheap and cheerful protection to the archive, which is very portable without requiring extra security software on the target machine, though it will not resist a determined effort to break it.
  • Data security: a data error in a filing system may lose a few files residing in the faulty sector, but can lose all the files in a ZIP archive.
  • Archive size: if you store a lot of large files in a ZIP archive, the archive size may become unmanageable; I would avoid any archives over 20GB.

You are very wise to back up in two places. As for what you back up, I suggest you do some tests as to what files give worthwhile compression: in my experience text files and document (.txt, .log, .doc, .docx, etc) files can save up to 90%, uncompressed multimedia files (.wav, .bmp, etc) around 50%, but compressed files (.mp*, .zip, some .pdf, installation files, etc) will give at most 2%, if that.

If you do zip your files, you may want to consider one file per archive. This will not give as good compression as multiple file archives, but will give reasonable data security.

For the record, I have used 7zip for years and find it excellent, and it has a command-line interface which is useful for batch operations.

AFH
  • 17,958
2

This is definitely and opinion based Q/A but, I would recommend software like WinRAR that can create a recovery record. Archives are great for a snapshot in time, ease of management, and tracking but large archives are susceptible to corruption.

You can hash the files, but this will not allow recovery; it will simply tell you the file(s) is not the original anymore. Archives with a recovery record can actually heal themselves.

Space savings for a large size consumer drive usually means lots of compressed data meaning you space savings will be very small so largest gain is ease of management and recovery; but you have to us an archive method that can actually do that. I am unaware of a long standing format other than WinRAR.

As for the device, a running NAS is much better than a cold stored HDD because a NAS can tell you if a drive is bad so you can take care of the problem where a cold stored HDD may be go from "good" to "bad" and you don't even know it. Cloud is probably your best option if your not tech savy. If you are tech savy, get a used LTO3/4 drive and tapes with a controller card. A prebuilt NAS can be a more expensive option after you calc in the upfront hardware and power cost over time.

Damon
  • 1,944