2

I compressed and encrypted (AES256) a folder with millions of JSONs using (in Ubuntu):

7za a -tzip -p -mem=AES256 json.zip json/

The compression seems to have gone fast, 4 hours or so. Now, I am decompressing/unencrypting the json.zip:

7za x json.zip

But it will take around a week. I noticed that it is using only one CPU

Is there a tool to decompress/unencrypted the files in parallel?

EDIT: Zip/Encryption uses all CPUs, but not unzip/decryption

I just tested both processes while observing the CPU usage.

  • 7za a -tzip -p -mem=AES256 json.zip json/ uses all CPUs
  • 7za x json.zip json/ uses only one

I tested this on the same machine to make sure it is not a configuration issue.

EDIT: Environment description

I need to transfer via network (ssh) 8M files compressed and encrypted with AES 256 from machine A to machine B. Here are the steps:

  1. The files on machine A live in a an encrypted (LUKS) SSD disk in a folder called json. I managed to compress them in ~4 hours with 7za a -tzip -p -mem=AES256 json.zip json/

  2. I transferred the json.zip to machine B via ssh

  3. I am un-compressing the files in machine B in a non-encrypted SSD disk with 7za x json.zip. It will take 7 days. I tested the disk and it works properly.

I am also considering to forget about the compression. I only need to transfer the files encrypted, compression just makes sense to keep smaller backups (which I will need). This will be a recurrent task at some point, so it will certainly become a problem.

toto_tico
  • 601

0 Answers0