Lazy ByteString : memory exploding in certain cases

Question

Below we have two seemingly functionally equivalent programs. For the first the memory remains constant, whereas for the second the memory explodes (using ghc 7.8.2 & bytestring-0.10.4.0 in Ubuntu 14.04 64-bit):

Non-exploding :

--NoExplode.hs
--ghc -O3 NoExplode.hs
module Main where
import Data.ByteString.Lazy as BL
import Data.ByteString.Lazy.Char8 as BLC

num = 1000000000
bytenull = BLC.pack ""

countDataPoint arg sum
  | arg == bytenull = sum
  | otherwise = countDataPoint (BL.tail arg) (sum+1)

test1 = BL.last $ BL.take num $ BLC.cycle $ BLC.pack "abc" 
test2 = countDataPoint (BL.take num $ BLC.cycle $ BLC.pack "abc") 0

main = do
  print test1
  print test2

Exploding :

--Explode.hs
--ghc -O3 Explode.hs
module Main where
import Data.ByteString.Lazy as BL
import Data.ByteString.Lazy.Char8 as BLC

num = 1000000000
bytenull = BLC.pack ""

countDataPoint arg sum
  | arg == bytenull = sum
  | otherwise = countDataPoint (BL.tail arg) (sum+1)

longByteStr = BL.take num $ BLC.cycle $ BLC.pack "abc" 

test1 = BL.last $ longByteStr
test2 = countDataPoint (BL.take num $ BLC.cycle $ BLC.pack "abc") 0

main = do
  print test1
  print test2

Additional details :

The difference is that inExplode.hs I have taken BL.take num $ BLC.cycle $ BLC.pack "abc" out of the definition of test1, and assigned it to its own value longByteStr.

Strangely if we comment out either print test1 or print test2 in Explode.hs (but obviously not both), then the program does not explode.

Is there a reason memory is exploding in Explode.hs and not in NoExplode.hs, and also why the exploding program (Explode.hs) requires both print test1 and print test2 in order to exlode?

score 4 · Accepted Answer · edited May 23 '17 at 10:33

4

Why ghc performs common expression elimination in one case, but not in the other? Who knows. Maybe common expressions where killed by inlining. Basically it depends on internal implementation.

Regarding -ddump-simp, see this SO question: Reading GHC Core

I reproduced it with ghc-7.8.2. It performs common expression elimination. You can check output of -ddump-simpl. So you actually creating one lazy bytestring.

In the first version you create two lazy bytestrings. print test1 forces the first one, but it is garbage collected on the fly because nobody else uses it. The same for print test2 -- it forces the second bytestring, and it is GC'ed on the fly.

In the second version you create one lazy bytestring. print test1 forces it, but it can't be GC'ed because it is needed for print test2. As a result, after the first print you have entire bytestring loaded into memory.

If you remove one print, the bytestring is GC'ed on the fly again. because it is not used anywhere else.

PS. "GC'ed on the fly" means: print takes the first chunk and outputs it to stdout. The chunk becomes available for GC. Then prints takes the second chunk, etc...

edited May 23 '17 at 10:33

Community

1
1

answered Jun 18 '14 at 15:54

Yuras

13,856
1
45
58

4

"In the second version you create one lazy bytestring. `print test1` forces it, but it can't be GC'ed because it is needed for `print test2`" : I am not sure I fully understand this. `longByteStr` is only used for `test1` and not `test2` – artella Jun 18 '14 at 16:13
@artella Hmm.. yes, I missed that, sorry. But I can't reproduce it with `ghc-7.6`. What compiler version are you using? – Yuras Jun 18 '14 at 16:23
Hi, I am using `ghc-7.8.2` on Ubuntu 64-bit and I have tested is quite a few times and can consistently reproduce it. – artella Jun 18 '14 at 16:29
I was using `7.6.3`, and I reproduced it with `7.8.2`. See the updated answer. – Yuras Jun 18 '14 at 16:35
Ah that's cunning. I was unaware of this optimization. I have indirectly verified this by changing `"abc"` to `"def"` in the expression for `test2` in `Explode.hs`, and in this case the program no longer explodes! Thanks – artella Jun 18 '14 at 16:42

Lazy ByteString : memory exploding in certain cases

1 Answers1