Akka Actor not terminating if an exception is thrown

Question

I am currently trying to get started with Akka and I am facing a weird problem. I've got the following code for my Actor:

class AkkaWorkerFT extends Actor {
  def receive = {
    case Work(n, c) if n < 0 => throw new Exception("Negative number")
    case Work(n, c) => self reply n.isProbablePrime(c);
  }
}

And this is how I start my workers:

val workers = Vector.fill(nrOfWorkers)(actorOf[AkkaWorkerFT].start());
val router = Routing.loadBalancerActor(SmallestMailboxFirstIterator(workers)).start()

And this is how I shut everything down:

futures.foreach( _.await )
router ! Broadcast(PoisonPill)
router ! PoisonPill

Now what happens is if I send the workers messages with n > 0 (no exception is thrown), everything works fine and the application shuts down properly. However, as soon as I send it a single message which results in an exception, the application does not terminate because there is still an actor running, but I can't figure out where it comes from.

In case it helps, this is the stack of the thread in question:

  Thread [akka:event-driven:dispatcher:event:handler-6] (Suspended) 
    Unsafe.park(boolean, long) line: not available [native method]  
    LockSupport.park(Object) line: 158  
    AbstractQueuedSynchronizer$ConditionObject.await() line: 1987   
    LinkedBlockingQueue<E>.take() line: 399 
    ThreadPoolExecutor.getTask() line: 947  
    ThreadPoolExecutor$Worker.run() line: 907   
    MonitorableThread(Thread).run() line: 680   
    MonitorableThread.run() line: 182

PS: The thread which is not terminating isn't any of the worker threads, because I've added a postStop callback, every one of them stops properly.

PPS: Actors.registry.shutdownAll workarounds the problem, but I think shutdownAll should only be used as a last resort, shouldn't it?

@ViktorKlang: But why does it remain, and how do I stop it properly? :) — fresskoma, May 30 '11 at 16:34
When should it be stopped? See the shutdown method here: http://akka.io/api/akka/1.1.2/#akka.event.EventHandler$ — Viktor Klang, May 30 '11 at 19:16
Yeah but the problem is I never start such a handler. It magically appears as soon as I send a message to my actor which triggers an exception, but I have no control over it. — fresskoma, May 30 '11 at 19:44
You start it when you're using the default event handler for the "logging" of Akka. It's configurable in the akka.conf — Viktor Klang, Jun 01 '11 at 04:58
Okay, but shouldn't it automatically shut down? It does when no error occurred. That behavior seems pretty weird and non-transparent at all to me. — fresskoma, Jun 01 '11 at 13:03
Well that was my original question: "When should it be stopped?" — Viktor Klang, Jun 03 '11 at 06:25
It should be stopped when all the other actors I've created are stopped too, that is, after all calculations are finished and everything is shut down (router, all actors). — fresskoma, Jun 03 '11 at 06:47
And in the case of the presence of a remote actor, how does it know if it's stopped or not? — Viktor Klang, Jun 03 '11 at 16:59
Aside from the fact that there are now remote actors, my problem is not that it is there or it is not there, my problem is that the actor exiting is dependent on whether an error occurs or not. — fresskoma, Jun 04 '11 at 12:11
@Pablo Fernandez: But why should I have to disable logging in order to have my application terminate properly? IMHO that is more of a workaround than a solution... — fresskoma, Nov 07 '11 at 14:12
Perhaps this question can be resolved by creating a ticket on `akka`? I'm just curious why this question has 40 upvotes in a place where the scala involvement is strong like SO and it didn't get an answer yet — Pablo Fernandez, Nov 07 '11 at 14:23
@x3ro: Just stumbled across this (when looking for the reason for nontermination). Have been able to verify that the remaining actor is indeed the `EventHandler`? (for me it was.) Sounded interesting, so I checked the code: For you the reason seems to be that throwing an exception will lead to `EventHandler.error` to be called (to print the stacktrace?).... I guess you will have to also throw a PoisonPill at the EventHandler to shut it down, some akka expert might have a better (nicer) solution. — subsub, Nov 08 '11 at 13:28

Arnon Rotem-Gal-Oz · Accepted Answer · 2013-10-02T16:35:07.070

The proper way to handle problems inside akka actors is not to throw an exception but rather to set supervisor hierarchies

"Throwing an exception in concurrent code (let’s assume we are using non-linked actors), will just simply blow up the thread that currently executes the actor.

There is no way to find out that things went wrong (apart from inspecting the stack trace). There is nothing you can do about it."

see Fault Tolerance Through Supervisor Hierarchies (1.2)

* note * the above is true for old versions of Akka (1.2) In newer versions (e.g. 2.2) you'd still set a supervisor hierarchy but it will trap Exceptions thrown by child processes. e.g.

class Child extends Actor {
    var state = 0
    def receive = {
      case ex: Exception ⇒ throw ex
      case x: Int        ⇒ state = x
      case "get"         ⇒ sender ! state
    }
  }

and in the supervisor:

class Supervisor extends Actor {
    import akka.actor.OneForOneStrategy
    import akka.actor.SupervisorStrategy._
    import scala.concurrent.duration._

    override val supervisorStrategy =
      OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
        case _: ArithmeticException      ⇒ Resume
        case _: NullPointerException     ⇒ Restart
        case _: IllegalArgumentException ⇒ Stop
        case _: Exception                ⇒ Escalate
      }

    def receive = {
      case p: Props ⇒ sender ! context.actorOf(p)
    }
  }

see Fault Tolerance Through Supervisor Hierarchies (2.2)

I had the same problem with exceptions: Thrown, displayed, but the actor processed further messages as if the exception hasn't occurred. So I switched to supervisor hierarchies. Exhausting to do so only because exceptions can't be handled otherwise. — Peter Schmitz, Nov 11 '11 at 18:41
This doesn't appear to be true in [more recent versions of Akka](http://doc.akka.io/docs/akka/2.2.1/scala/fault-tolerance.html), where throwing exceptions is exactly what supervisors supervise. — nornagon, Oct 02 '13 at 15:03

score 2 · Answer 2 · answered Jan 05 '12 at 19:36

Turning off the logging to make sure things terminate, as proposed by Viktor, is a bit strange. What you can do instead is:

EventHandler.shutdown()

that cleanly shuts down all the (logger) listeners that keep the world running after the exception:

def shutdown() {
  foreachListener(_.stop())
  EventHandlerDispatcher.shutdown()
}

score -3 · Answer 3 · edited Jul 29 '12 at 20:24

-3

Turn of the logger in the akka.conf

edited Jul 29 '12 at 20:24

burning_LEGION

13,246
8
40
52

answered Nov 09 '11 at 09:17

andersoyvind

267
3
12

Akka Actor not terminating if an exception is thrown

3 Answers3