2

I am developing a spark application. To test it locally I want to run sbt run. This requires the dependencies to be available locally. But I also want to use sbt assembly to generate a jar which can be used for spark-submit that jar should only include some of the dependencies e.g. not include the spark dependencies.

When I mark a dependency in build.sbt as % "provided" it no longer is available via sbt run

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
  • This question has 2 subquestions - you should split it to two separate questions to make it easier to answer – T. Gawęda Nov 24 '16 at 09:18
  • 1
    ok I wil do that. Here is question 2: http://stackoverflow.com/questions/40782435/sbt-switch-dependencies-for-runtime – Georg Heiler Nov 24 '16 at 09:25

1 Answers1

2

If the case to exclude jars in assembly use in main built.sbt

excludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter {f =>
     f.data.getName.contains("spark"),
     f.data.getName.startsWith("jar_name")
  }
}
FaigB
  • 2,271
  • 1
  • 13
  • 22
  • 1
    but still this does not exclude transitive dependencies for spark e.g. I am facing problems with kryo .. how can I exclude all "transitive" dependencies of a jar as well? – Georg Heiler Nov 24 '16 at 10:06
  • I want to add http://stackoverflow.com/questions/18838944/how-to-add-provided-dependencies-back-to-run-test-tasks-classpath/21803413#21803413 is a nicer solution for my specific usecase with spark. – Georg Heiler Nov 30 '16 at 15:46