Parallel compression C#

Question

Is it possible to optimize this code using Parallel.Foreach or something?

using (var zipStream = new ZipOutputStream(OpenWriteArchive()))
{
    zipStream.CompressionLevel = CompressionLevel.Level9;    
    foreach (var document in docuemnts)
    {
        zipStream.PutNextEntry(GetZipEntryName(type));    
        using (var targetStream = new MemoryStream()) // document stream
        {
            DocumentHelper.SaveDocument(document.Value, targetStream, type);    
            targetStream.Position = 0; targetStream.CopyTo(zipStream);
        }    
        GC.Collect();
    };
}

The problem is DotNetZip's and SharpZipLib's ZipOutputStream doesn't support position changing or seeking.

Writing to zip stream from multiple threads leads to error. It's also impossible to accumulate result streams into ConcurrentStack beacuse application can work with 1000+ documents and should to compress and save streams into the cloud on the fly.

Is there any way to solve this?

Do you *have* to zip all the files into one and the same archive? — Timothy Groote, Jul 08 '13 at 09:04
Yes, all entries (documents) should be into one and the same archive — Kirill Karahainko, Jul 08 '13 at 09:07
If the ZipOutputStream *did* support seeking and position changing and you managed to implement a marshalling system that would allow multiple threads to write in it simultaneously, it would still be your bottleneck. A marshalling system like that would probably also cause more overhead than you could gain from parallel processing. — Timothy Groote, Jul 08 '13 at 09:07

score 1 · Accepted Answer · answered Jul 09 '13 at 13:43

Resolved by using of ProducerConsumerQueue (Producer-Consumer pattern).

using (var queue = new ProducerConsumerQueue<byte[]>(HandlerDelegate))
{
    Parallel.ForEach(documents, document =>
    {
        using (var documentStream = new MemoryStream())
        {
            // saving document here ...

            queue.EnqueueTask(documentStream.ToArray());
        }
    });
}

protected void HandlerDelegate(byte[] content)
{
    ZipOutputStream.PutNextEntry(Guid.NewGuid() + ".pdf");

    using (var stream = new MemoryStream(content))
    {
        stream.Position = 0; stream.CopyTo(ZipOutputStream);
    }
}

score 0 · Answer 2 · answered Jul 08 '13 at 09:30

Try to decleare zipstream inside parallel foreach, like:

Parallel.ForEach(docuemnts, (document) =>
            {
                using (var zipStream = new ZipOutputStream(OpenWriteArchive()))
                {
                    zipStream.CompressionLevel = CompressionLevel.Level9;
                    zipStream.PutNextEntry(GetZipEntryName(type));
                    using (var targetStream = new MemoryStream()) // document stream
                    {
                        DocumentHelper.SaveDocument(document.Value, targetStream, type);
                        targetStream.Position = 0; targetStream.CopyTo(zipStream);
                    }
                    GC.Collect();
                }
            });

Bye!

All documents should be in the same archive – Kirill Karahainko Jul 08 '13 at 09:56 — Kirill Karahainko, Jul 08 '13 at 09:56

Parallel compression C#

2 Answers2

Linked