Handling sum encoding in Streaming libraries

Question

The motivation behind this question is this scenario - we have a stream of values which are represented by a Sum encoding. Let us assume Either ByteString ByteString where we represent streams of bytes in error and good states respectively. Now, we have another function which can compress the ByteString stream. Is it possible to run this function on Either ByteString ByteString input stream, and compress either one (not just Right but also Left in case Left is yielded instead of Right). A compress function type signature is below (I am using Streaming library):

compress ::  MonadIO m 
         =>  Int 
         -- ^ Compression level.
         -> Stream (Of ByteString) m r
         -> Stream (Of ByteString) m r

Our input stream is of type Stream (Of (Either ByteString ByteString)) m r. So, is there some kind of transformer function that can run compress on input stream, and output a stream of type Stream (Of (Either ByteString ByteString)) m r where both are compressed.

It seems to me that I should write a custom compress instead, let us say eitherCompress as follows:

eitherCompress :: MonadIO m 
             =>  Int 
             -- ^ Compression level.
             -> Stream (Of (Either ByteString ByteString)) m r
             -> Stream (Of (Either ByteString ByteString)) m r

Is that correct? If that is the case, what is a good way to write eitherCompress using the function below from zstd library:

compress :: Int 
         -- ^ Compression level. Must be >= 1 and <= maxCLevel.
         -> IO Result

I have written stream producers using yield, but I have implemented them for simple cases where the input is just a source, not a stream. Will very much appreciate help with this problem.

danidiaz · Answer 1 · 2018-03-13T23:31:24.067

A common trick to solve these cases is to put each branch of the sum in different monadic layers (so there will be two streaming layers) manipulate each layer separately, and then either consume them separately or re-join them in a single layer.

First, two auxiliary functions that use maps to convert to and from the Sum composition of functors:

toSum :: Monad m 
      => Stream (Of (Either ByteString ByteString)) m r 
      -> Stream (Sum (Of ByteString) (Of ByteString)) m r
toSum = maps $ \(eitherBytes :> x) -> 
    case eitherBytes of
        Left bytes -> InL (bytes :> x)
        Right bytes -> InR (bytes :> x)

fromSum :: Monad m 
        => Stream (Sum (Of ByteString) (Of ByteString)) m r 
        -> Stream (Of (Either ByteString ByteString)) m r
fromSum = maps $ \eitherBytes ->
    case eitherBytes of
        InL (bytes :> x) -> Left bytes :> x
        InR (bytes :> x) -> Right bytes :> x

We do this to be able to use the separate and unseparate functions.

The actual compression function would be:

eitherCompress :: MonadIO m 
               => Int 
               -> Stream (Of (Either ByteString ByteString)) m r 
               -> Stream (Of (Either ByteString ByteString)) m r
eitherCompress level =
     fromSum . unseparate . hoist (compress level) . compress level . separate . toSum

hoist is used to work over the monadic layer below the topmost one.

Perfect, that is what I was looking for! I saw the `sum` encoding in the documentation but haven't been able to piece together the various pieces until you answered it by pointing out `toSum` and `fromSum` approach. — Sal, Mar 13 '18 at 21:47
question - if `compress` is specialized to a concrete return type (.e.g.,`Stream (Of ByteString) m ()` - instead of `r` we now have `()`), then `hoist` won't work because of `forall` quantifier. In that case, what would be a replacement for `hoist`? — Sal, Mar 14 '18 at 00:47

Handling sum encoding in Streaming libraries

1 Answers1