Take garbage collection. When it's time to collect garbage in other
  languages, the entire system has to stop while the garbage collector
  runs. This approach is perfectly fine if your computer program is
  supposed to run once, write some output, and then quit. But in
  long-running applications, such as desktop, mobile, or server
  programs, this strategy results in occasionally frozen UIs and slow
  response times. Erlang programs, on the other hand, can have thousands
  of independent heaps which are garbage-collected separately; in this
  way, the performance penalty of garbage collection is spread out over
  time, and so a long-running application will not mysteriously stop
  responding from time to time while the garbage collector runs.
Evan Miller, creator of the popular Chicago Boss framework.
So I believe erlang garbage collects concurrently, that is, the various heaps are garbage collected independently of one another. Whether there is any parallelization depends on whether your node is running on multiple cores or not, but if so then the garbage collection is done in parallel, yes.