1

I have mixed Java/Scala project. There are Quartz jobs that are implemented in Java and use some Scala classes. These classes should use the same SparkContext instance, so I implemented something that should be singleton and looks like this:

object SparkContextLoader {
    var hasSC = false
    var sc:Any = 0
    def getSC(workers):SparkContext={
    if (!hasSC) {
        val sparkConf = new SparkConf().setMaster("local[" + workers + "]").setAppName("SparkApp")
        sc = new SparkContext(sparkConf)
        hasSC = true
    }
    return sc.asInstanceOf[SparkContext]
}

Calling SparkContextLoader from two different jobs always creates a new SparkContext instance which is not allowed.

Why Scala object doesn't behave like singleton?

Aaditya Gavandalkar
  • 814
  • 5
  • 17
zoran jeremic
  • 2,046
  • 4
  • 21
  • 47
  • are your quartz jobs written in java? and did you review how to reference scala objects in java code? – emran Jan 10 '16 at 06:24
  • 1
    also you are using a var instead of a val for the SparkContext – emran Jan 10 '16 at 06:26
  • 2
    Possible duplicate of [How can I use a Scala singleton object in Java?](http://stackoverflow.com/questions/12284028/how-can-i-use-a-scala-singleton-object-in-java) – 4e6 Jan 10 '16 at 08:14
  • @4e6 It's not duplicate with this. If you check my post, it's I'm not using Scala singleton object from Java. Java jobs create instances of some Scala classes and call a function on that classes. These scala classes use Scala object that should behave as Singleton, i.e. to return one SparkContext instance for the whole application. – zoran jeremic Jan 10 '16 at 16:45
  • As suggested by @heavyhorse the problem was that I was using var instead of val for SparkContext – zoran jeremic Jan 10 '16 at 16:47

1 Answers1

2

Your code is overly complicated. If the "two different jobs" are different threads then all you need is:

object SparkContextLoader {

  val sparkConf = new SparkConf().setMaster("local[" + workers + "]").setAppName("SparkApp")

  val sc = new SparkContext(sparkConf)
}

You can then access these vals using the answer to this question.

If the "two different jobs" are different java applications then it doesn't seem that there is a way to share a singleton across both of them.

Ramón J Romero y Vigil
  • 17,373
  • 7
  • 77
  • 125
  • You were right. Using var instead of val for SparkContext was the problem. I expected scala object would always behave like Singleton, but it's obviously not the case. – zoran jeremic Jan 10 '16 at 16:49