Spark 2.4.3 - Scala 2.12.3 - object apache is not a member of package org

Question

I recently updated to Spark 2.4.3 and Scala 2.12.3 (from Spark 2.0.0), and I having issues compiling very simple code (load and show).

My build.sbt with sbt 1.2.8 is:

name := "my-program"

version := "1.0"

scalaVersion := "2.12"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.3"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.4.3"

I am developing with Scala for Eclipse, and the jars spark-core_2.12_2.4.3.jar, spark-mllib_2.12_2.4.3.jar, and spark-sql_2.12_2.4.3.jar are linked to my build path (Eclipse shows no error).

I updated Spark, scala, and sbt with homebrew. I don't know if that messes up with how sbt finds the jars?

I tried sbt clean package and sbt package many times, but all I get is:

[error] /Users/me/myproject/src/main/scala/Analysis.scala:5:12: object apache is not a member of package org
[error] import org.apache.spark.sql.SparkSession

I am out of ideas of what to try.

Analysis.scala:

package main.scala

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._


object Analysis {


   def main(args: Array[String]) {

     // Start Spark session
      val spark = SparkSession.builder().getOrCreate()
      import spark.implicits._
      // Reduce verbosity of output when running in console
      spark.sparkContext.setLogLevel("WARN")

      val df =   spark.read   
      // Format read
      .format("com.databricks.spark.csv")
      .option("header", "true")
      .option("parserLib", "UNIVOCITY")
      .option("inferSchema", "true")
      .json("data/transaction.txt")

      df.printSchema()
   }  

}

Are you creating complete jar file and then executing it? If yes, can you look into it if you have the dependencies packaged in JAR file? — Piyush Patel, Aug 20 '19 at 18:30
I have never codes in Scala on Eclipse but ... are you sure about `package main.scala` ? That is odd. — Catalina Chircu, Aug 20 '19 at 19:01
Yes, I am sure. These are equivalent to subfolders. src/main/scala if you will. — Béatrice Moissinac, Aug 20 '19 at 19:02
I dont think itexactly means that. src/main/scala is the default folder structure for sbt or maven projects I believe and then inside that you can create packages like com.piyushpatel.something. Then the complete path will be src/main/scala/com/piyushpatel/something/Main.scala — Piyush Patel, Aug 21 '19 at 13:37

score 1 · Answer 1 · answered Aug 20 '19 at 18:24

1

In your build.sbt specify full version of scala (2.12.8 instead of 2.12). It should fix it (it did in my case).

scalaVersion := "2.12.8"

I am not sure why this happens with sbt.

answered Aug 20 '19 at 18:24

shanmuga

4,329
2
21
35

score 0 · Answer 2 · answered Aug 20 '19 at 17:59

0

It must be a problem of compatibility between Spark and Scala versions. See this link.

Try Scala version 2.12.8 + Spark version 2.4.0. It worked for me.

answered Aug 20 '19 at 17:59

Catalina Chircu

1,506
2
8
19

I am at a complete loss. I installed Spark 2.4.0, and scala 2.12.8 - I updated my build.sbt but I still get the same error! – Béatrice Moissinac Aug 20 '19 at 18:40
I confess I use IntelijIdea, and not Eclipse like you. That might make a difference too. – Catalina Chircu Aug 20 '19 at 18:54
I have another Eclipse workspace with a Spark project that works just fine, even with the update. I have no idea why this new workspace/project doesn't! Same build.sbt one gets the error, the other doesn't. Been working on this all f****g day! *SCREAMS!* – Béatrice Moissinac Aug 20 '19 at 19:01
OMFG... The build.sbt was in the wrong folder... project/src/build.sbt instead of project/build.sbt So I never was calling my dependencies... Please kill me now... – Béatrice Moissinac Aug 20 '19 at 19:04
So it works now? By the way I tried it on my computer, I just had to add `.master("local")` when creating `SparkSession`. – Catalina Chircu Aug 20 '19 at 19:12
I thought that since Spark 2.2 we didn't need to set the master anymore. Either way, it works now. I need a beer -_- – Béatrice Moissinac Aug 20 '19 at 19:47
Nope. I've got an error for not setting the master. Enjoy your beer! – Catalina Chircu Aug 20 '19 at 20:02

score 0 · Accepted Answer · answered Aug 20 '19 at 19:06

0

The build.sbt was not in the project folder (/Users/me/myproject/), but in the src folder (/Users/me/myproject/src) so when I did sbt package, it never found my build.sbt, and therefore had dependencies problem.

answered Aug 20 '19 at 19:06

Béatrice Moissinac

934
2
16
41

Spark 2.4.3 - Scala 2.12.3 - object apache is not a member of package org

3 Answers3