Clojure Parallel Mapping and Infinite Sequences

Question

Let's say I define the sequence of all natural numbers in the following way:

(def naturals (iterate inc 0))

I also define a function mapping the naturals to nil that takes a while to compute like so:

(defn hard-comp [_] (Thread/sleep 500))

Note the computation time to evaulate the following s-expressions as measured by clojure.core/time.

(dorun (map hard-comp (range 30))) ; 15010.367496 msecs

(dorun (pmap hard-comp (range 30))) ; 537.044554 msecs

(dorun (map hard-comp (doall (take 30 naturals))))) ; 15009.488499 msecs

(dorun (pmap hard-comp (doall (take 30 naturals)))) ; 3004.499013 msecs

(doall (take 30 naturals)) ; 0.385724 msecs

(range 30) ; 0.159374 msecs

pmap is ~6 times faster when called with an explicit range than with a section of the naturals.

Since (= (range 30) (take 30 naturals)) returns true and both objects are of type clojure.lang.LazySeq, and clojure evaulates all the arguments to a function before calling the function, how can the above timing details be explained?

Answer 1

My guess is that it's due to this:

user> (chunked-seq? (seq (range 30)))
true
user> (chunked-seq? (seq (take 30 naturals)))
false
user> (class (next (range 30)))
clojure.lang.ChunkedCons
user> (class (next (take 30 naturals)))
clojure.lang.Cons

Try this:

user> (defn hard-comp [x] (println x) (Thread/sleep 500))
#'user/hard-comp
user> (time (dorun (pmap hard-comp (range 100))))

Note that it jumps 32 items at a time. That's how many elements are grabbed per chunk for a range. Chunked seqs pre-evaluate a bunch of items ahead of time to increase performance. In this case it looks like pmap chunkily spawns 32 threads as soon as you try to grab even one element from the range.

You could always stuff your naturals into a vector to get chunking behavior.

user> (time (dorun (pmap hard-comp (range 100))))
"Elapsed time: 2004.680192 msecs"
user> (time (dorun (pmap hard-comp (vec (take 100 naturals)))))
"Elapsed time: 2005.887754 msecs"

(Note that the time is approximately 4 x 500 ms, 4 being how many chunks of 32 it takes to get to 100.)

On the other hand, you might not want chunking behavior. 32 threads at a time is a lot. See this question for examples of how to un-chunkify a seq.