sbt-assembly shading error with shapeless in ubuntu

Question

I’m having a weird issue with sbt-assembly if anyone could help

When trying to create fat jar to deploy to Spark with shading applied to shapeless libraries, I am seeing some classes not being renamed when ran in an Ubuntu machine while everything gets renamed fine when the sbt assembly is ran in Mac.

Here’s the shading config

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadedshapeless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" % "pureconfig_2.11" % "0.7.0")
    .inProject)

When ran in Mac, these classes are renamed for pattern shapeless/Generic*

Renamed shapeless/Generic$.class -> shadedshapeless/Generic$.class
Renamed shapeless/Generic.class -> shadedshapeless/Generic.class
Renamed shapeless/Generic1$.class -> shadedshapeless/Generic1$.class
Renamed shapeless/Generic1$class.class -> shadedshapeless/Generic1$class.class
Renamed shapeless/Generic1.class -> shadedshapeless/Generic1.class
Renamed shapeless/Generic10$class.class -> shadedshapeless/Generic10$class.class
Renamed shapeless/Generic10.class -> shadedshapeless/Generic10.class
Renamed shapeless/Generic1Macros$$anonfun$1.class -> shadedshapeless/Generic1Macros$$anonfun$1.class
Renamed shapeless/Generic1Macros$$anonfun$2.class -> shadedshapeless/Generic1Macros$$anonfun$2.class
Renamed shapeless/Generic1Macros.class -> shadedshapeless/Generic1Macros.class
Renamed shapeless/GenericMacros$$anonfun$23.class -> shadedshapeless/GenericMacros$$anonfun$23.class
Renamed shapeless/GenericMacros.class -> shadedshapeless/GenericMacros.class

but when ran in Ubuntu, for the pattern shapless/Generic* only these things are renamed

Renamed shapeless/GenericMacros$$anonfun$23.class -> shadedshapeless/GenericMacros$$anonfun$23.class
Renamed shapeless/Generic1Macros$$anonfun$1.class -> shadedshapeless/Generic1Macros$$anonfun$1.class
Renamed shapeless/Generic1$.class -> shadedshapeless/Generic1$.class
Renamed shapeless/Generic1.class -> shadedshapeless/Generic1.class

I chose the pattern shapeless/Generic* coz when am providing the fat jar (produced in Ubuntu) to spark-submit then am getting the error right away (probably coming from pureconfig)

Exception in thread "main" java.lang.NoClassDefFoundError: shadedshapeless/Generic

No error occurs when fat jar produced in Mac is being fed to spark-submit

I have the same issue on windows sub-system vs linux. Some classes inside shapeless is not renamed. Have you found the answer? — Minh Thai, Dec 11 '18 at 02:55

score 0 · Answer 1 · answered Jun 21 '17 at 20:28

I'm not sure why on Ubuntu shading works differently from MacOs but one issue that I see is that you are shading only part of shapeless. I don't think this is a good idea and can give you problems because you'll mix different versions of shapeless. My suggestion is to try shading shapeless entirely for PureConfig. Just add

assemblyShadeRules in assembly := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

to your sbt file. This solution has been tested by me on Ubuntu 16.04 with PureConfig 7.2 and Spark 2.1.0. See the FAQ section of PureConfig about this issue.

sbt-assembly shading error with shapeless in ubuntu

1 Answers1