2

While checking the throughput of spray api.

Scenario: 25 concurrent users

Os: Free BSD

Memory: 2GB

No Of Cores: 2

At around 13 concurrent users i was getting the following error.

[ERROR] [06/29/2015 05:01:56.407] [default-akka.actor.default-dispatcher-2]     [ActorSystem(default)] Uncaught error from thread [default-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
    at scala.concurrent.forkjoin.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1795)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:117)

Akka and Spray Conf changes from default:

akka{
    tcp{
        register-timeout = 20s
    }
}

spray.can {
    request-timeout = 30 s
    bind-timeout = 30s
    unbind-timeout = 5s
    registration-timeout = 30s
}

http.spray.can {
    server{
        pipelining-limit = 50
    }
}

What is causing the OutOfMemmoryError. The exception is thrown from the router actor

Rahul Dev
  • 387
  • 1
  • 4
  • 12

1 Answers1

8

can't read mind, but probably you do blocking (Await.result or similar) inside actors. ForkJoinPool creates a new thread for every blocked one automatically. So if you have a long-time blocks count_of_threads == count_of_requests (+ every thread holds the references iside call stack), which eventually causes OutOfMemory.

See, Blocking Needs Careful Management

P.S. Here you may find why Await.result (which uses scala.concurrent.blocking inside) leads to unmanageable creation of threads in ForkJoinPool(even regardless of maxParallelism).

Or you creating a lot of ActorSystems, same page of akka documentation states:

An ActorSystem is a heavyweight structure that will allocate 1…N Threads, so create one per logical application.

aht
  • 117
  • 1
  • 7
dk14
  • 22,206
  • 4
  • 51
  • 88
  • I was making Await.result inside actors. Now i removed that and made it to Future. But other actors have blocking calls such as REST calls. Will that be the issue. – Rahul Dev Jun 29 '15 at 12:25
  • Had 2 actor systems in the application. Now made it to one and seems to be working. Can you please explain the change in behavior. – Rahul Dev Jun 30 '15 at 05:09
  • As I said, I can't read mind :) - less actor systems - less memory, so it could help a bit. But two actor systems isn't enough to be a direct reason of this exception (maybe you had "`def`" factory with a lot of actor systems - then it could be a problem). Are you sure that ActorSystem removing helped or it actually was removing of `Await.result`?Answering your first question - blocking REST calls probably won't cause this exception (however it's still bad) because their blocking is unmanagable and `ForkJoinPool` knows nothing about them so it won't create new thread for. – dk14 Jun 30 '15 at 05:23
  • Got it. I was creating an actor system inside an actor, such that every time that actor is called it will create an actor system. :) Thanks for your quick help. – Rahul Dev Jun 30 '15 at 07:47
  • Np. Added this to the answer to make it correct one. – dk14 Jun 30 '15 at 08:33