4

What I need to do is compress 64KB chunks of a file as they're coming in from an external service and then stream that chunk of compressed data to the browser all in real time. Xceed Real-Time is pretty awesome in how you can set the header of the file and then piece together the multiple 64KB chunks into a whole zipped file on the client. This works great and you can easily handle multiple files by putting it in a loop.

My question is this: Is there a free alternative compression utility that can match the sophistication of Xceed? I need something that can compress and stream parts of a file to the browser. Pretty much all of the free alternatives I've seen require having access to the entire file(s) before compression can take place. Xceed is great, but it's expensive. Just wondering if there's a free alternative out there that can accomplish this. Thanks!

TheDude
  • 1,421
  • 4
  • 29
  • 54

2 Answers2

9

What you are looking for is stream compression approach. Good news is you have quite a lot of options. Please also note you have to think about memory because some solutions (dictionary-based) might require a lot of memory, so test and tune.

In order to keep my answer short I would suggest to give LZ4 a try.

LZ4 - http://code.google.com/p/lz4/

LZ4 is a very fast lossless compression algorithm, providing compression speed at 300 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in GB/s per core, typically reaching RAM speed limits on multi-core systems.

Benchmark: 

Name            Ratio  C.speed D.speed
                        MB/s    MB/s
LZ4 (r97)       2.084    410    1810
LZO 2.06        2.106    409     600
QuickLZ 1.5.1b6 2.237    373     420
Snappy 1.1.0    2.091    323    1070
LZF             2.077    270     570
zlib 1.2.8 -1   2.730     65     280
LZ4 HC (r97)    2.720     25    2040
zlib 1.2.8 -6   3.099     21     300

There are a lot of implementations available, please check: http://code.google.com/p/lz4/

LZO - http://lzo-net.sourceforge.net/

LZO.Net brings the power of Markus "FXJ" Oberhumer's great LZO compression library (V1.08) to .Net. It wraps the access to the native DLL with a small C# class maintaining the raw speed of the ANSI-C library.

Snappy - https://code.google.com/p/snappy/

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression.

Two C# native implementations are available:

QuickLZ - http://www.quicklz.com/

QuickLZ is the world's fastest compression library, reaching 308 Mbyte/s per core. It can be used under a commercial license if such has been acquired or under GPL 1, 2 or 3 where anything released into public must be open source.

Native implementations - QuickLZ C#

So far, only a subset of the library has been ported, 
namely the setting:

QLZ_COMPRESSION_LEVEL = 1 or 3 
QLZ_STREAMING_BUFFER = 0 
QLZ_MEMORY_SAFE = 0
Renat Gilmanov
  • 17,735
  • 5
  • 39
  • 56
  • I would need a c# implementation of the library. Do you have an example in how I could use FastLZ to create headers for files on the fly and then tag the rest of the chunks to that header? Xceed has this functionality built in. Thanks! – TheDude Aug 07 '13 at 04:56
  • 1
    I've updated the answer. I did remove FastLZ, since it doesn't suit you well. So you have 4 options: LZ4, LZO, Snappy, QuickLZ (top of the benchmark). LZO is a wrapper. QuickLZ C# implementation has some limitations. Anyway, you are good to go with LZ4 or Snappy. – Renat Gilmanov Aug 07 '13 at 10:06
  • Can LZ4 decompress zipped files? – zer0ne Apr 06 '14 at 08:50
2

Check out SharpCompress, written specifically for purposes like yours. I use it and works perfect for my solution. We used Xceed before our re-write.

http://sharpcompress.codeplex.com/

SharpCompress is a compression library for .NET/Mono/Silverlight/WP7 that can unrar, un7zip, unzip, untar unbzip2 and ungzip with forward-only reading and file random access APIs. Write support for zip/tar/bzip2/gzip is implemented.

The major feature is support for non-seekable streams so large files can be processed on the fly (i.e. download stream).

Scott Mitchell
  • 125
  • 1
  • 6
  • Do you, by any chance, have an example in how to use SharpCompress to compress 64KB chunks in real-time? With Xceed, there's built in functionality to create a header for each file and then the 64KB chunks are tagged to that file header. Does SharpCompress have similar functionality? Would appreciate an example! – TheDude Aug 07 '13 at 04:54
  • Thanks for this. The link is no longer valid. Try at https://www.nuget.org/packages/sharpcompress/ – Shoom Kloom May 03 '22 at 05:59