4

I have 37 GiB File which was created by tar + xz. I have a server with 6 cores, so for maximum compression speed I want to use all my 6 cores using this command

XZ_OPT='-T0 -9 –memory=75%' tar -cJf mydir.tar.xz mydir

And it was pretty fast. What can I do to make decompress using all the cores?

tar -xvf mydir.tar.xz

Uses only 1 core for decompression

Sathyajith Bhat
  • 62,374
alanyalilar
  • 41

2 Answers2

2

Normally, compression is the bottleneck, such that threaded decompression is not supported by most compression libs. To quote the xz man-pages:

Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't even if --block-size=size is used.

NaN
  • 121
2

As of xz 5.4.1 (December 2022), there is support for parallel decompression with -T0. However, the tar file format necessitates a sequential read, so some buffering up to thread_count * block_size would be needed. I am not sure whether xz has that kind of buffering in place to support piping.

Mingye Wang
  • 193