Hikari NoSuchMethodError on AWS EMR/Spark

Question

I am trying to upgrade EMR from 5.13to 5.35 using spark-2.4.8. The jar I'm trying to use has a dependency on HikariCP:4.0.3 which is called to set the db pool-config setKeepaliveTime. While I can run my job fine on my local machine, it bombs out in EMR-5.35 with the following error:

java.lang.NoSuchMethodError: com.zaxxer.hikari.HikariConfig.setKeepaliveTime(J)

The problem is, in runtime, the HikariConfig is being loaded from file:/usr/lib/spark/jars/HikariCP-java7-2.4.12.jar instead of what was provided as a dependency in my custom/fat jar. The workaround right now is to remove that jar, but is there an elegant way to know where that jar is coming from just on the EMR and how could we remove that on start-up?

Answer 1

Just in case, anyone else faces this, the fix was shading (Process that allows renaming packages on the Uber jar), I basically had to make sure the dependency I use doesn't get overridden with the one that's stale in EMR-5.35.0. It looked something like the below:

assembly / assemblyShadeRules := Seq(
  ShadeRule
    .rename("com.zaxxer.hikari.**" -> "x_hikari_conf.@1")
    .inLibrary("x" % "y" % z)
    .inProject
  )

And that was pretty much it, after the above lines were put in and the new jar was created, it worked like a charm.

More on shading can be read here