42

Does anyone know how to get cabal install to exploit parallelism? I'm compiling with GHC, and while I don't know if GHC itself can do parallel builds, surely cabal install could run multiple compilations in parallel, no? At least for distinct, independent packages?

Does anyone know if it is possible and how to do it?

imz -- Ivan Zakharyaschev
  • 4,921
  • 6
  • 53
  • 104
Norman Ramsey
  • 198,648
  • 61
  • 360
  • 533
  • What's the motivation for this? Isn't the main bottleneck for cabal the network transfers? Also, couldn't build errors bump into each other, if there are shared dependencies? – amindfv Aug 30 '11 at 02:58
  • 2
    @amindfv: for me, the bottleneck is compiling dozens of haskell source files. And about dependencies, like parallel make, parallel `cabal install` would have to respect them. But there is still parallelism available to be had. – Norman Ramsey Sep 01 '11 at 21:35
  • there's also an open [stack issue](https://github.com/commercialhaskell/stack/issues/644)... (if you're using stack instead of cabal-install) – mb21 Dec 24 '16 at 12:45

3 Answers3

51

I'm the one who was working on this Summer of Code project. The patches have been sent to Duncan, but he hasn't reviewed them yet. Note that my code works at the package granularity, so you won't get any speedup when building a single package. I'm currently working on a parallel wrapper around ghc --make, which will solve this problem (I hope to get it merged into the mainline cabal-install eventually).

Update (Feb. 2012): Duncan has reviewed my patches, I need to incorporate his feedback and resubmit them. I hope to get this done before the end of this month.

Update (Apr. 2012): I've updated my patches in response to Duncan's comments. The new code is a bit slower, but requires much less changes to the Cabal library.

Update (Jun. 2012): Duncan Coutts just merged the parallel branch into Cabal HEAD. Parallel install will be available in the next cabal-install release.

Update (Oct. 2012): cabal-install1.16.0 has just been released. This is the first official release that includes my parallel patches.

Mikhail Glushenkov
  • 14,928
  • 3
  • 52
  • 65
  • That is fantastic! Can you give us a short summary about what the final patch can do? Are normal/profiling compilations parallel? Do single packages now use multiple cores? Thank you. – nh2 Jul 28 '12 at 21:17
  • And what is the state of `ghc-parmake`? – nh2 Jul 28 '12 at 21:50
  • 1
    @nh2 Maybe I should write a blog post about the current status of parallel install. For now, you can follow the discussion [here](https://github.com/haskell/cabal/issues/976). Short answer: add a line 'jobs: n' (where n is a positive integer) to your `~/.cabal/config`; this makes `cabal install` use n threads. This is only package-level parallelism; modules and components are not built in parallel. – Mikhail Glushenkov Jul 28 '12 at 22:22
  • @nh2 Regarding `ghc-parmake`, it works in some cases, but there are known bugs (can't build libs for profiling, can't build large projects such as uhc, etc). I'm working on another Cabal-related project (sandboxes), so I don't have time to spend on `ghc-parmake` or parallel `cabal build` right now. Maybe after the end of this GSoC. – Mikhail Glushenkov Jul 28 '12 at 22:30
  • I tried the cabal HEAD today (with GHC 7.4.1, and there was a type error, which I solved by deleting code, hehe), and it works pretty well! Will this be in the next Haskell platform? – nh2 Sep 20 '12 at 01:26
  • If the next Haskell Platform will ship with cabal-install 0.16, then yes. – Mikhail Glushenkov Sep 20 '12 at 10:08
  • 1
    @nh2 The next Haskell Platform (2012.4.0.0) will use cabal-install 0.14 and GHC 7.4.2. I expect the 2013.2.0.0 release to ship with cabal-install 1.16 and a more recent GHC. – Mikhail Glushenkov Oct 03 '12 at 16:42
  • Great, looking forward to that! – nh2 Oct 05 '12 at 12:02
  • With my recent `cabal update`, I got `Note: there is a new version of cabal-install available`, so a `cabal install cabal-install` now gave me 0.16, even on the Haskell Platform 2012. – nh2 Dec 06 '12 at 16:41
38

Completing Mikhail Glushenkov's answer to document the usage a bit:

As of cabal 1.16.0, you can now use

cabal install -j [pkgs…]

This defaults to one job per core. It also gives much cleaner output.

You can make parallel installs the default with:

echo "jobs: $(getconf _NPROCESSORS_ONLN)" >> ~/.cabal/config

Or (cabal-install 1.18+):

echo 'jobs: $ncpus' >> ~/.cabal/config

Get the latest cabal-install with:

cabal update
cabal install cabal-install --bindir ~/bin --upgrade-dependencies
Tobu
  • 24,771
  • 4
  • 91
  • 98
  • 4
    As of [Cabal 1.18](https://github.com/haskell/cabal/commit/99264248db78835b308232943038e6cbce4e72a4) you can set `jobs` setting to `$ncpus` or a number of jobs to run. – akamch Sep 30 '13 at 10:27
17

There was a Google Summer of Code project this summer to parallelize cabal-install. While it hasn't been merged into the mainline yet, the linked article provides instructions for grabbing the source and building it yourself.

hammar
  • 138,522
  • 17
  • 304
  • 385
  • @donatello: See the update in Mikhail's answer below. I'll see if I can get the link fixed. – hammar Feb 15 '12 at 20:15