Scala Script: Explain this class cast error to ScalaClassLoader on Windows

Question

Consider following scala script:

import scala.reflect.internal.util.ScalaClassLoader

object Test {
  def main(args: Array[String]) {
    val classloaderForScalaLibrary = classOf[ScalaClassLoader.URLClassLoader].getClassLoader
    println(classloaderForScalaLibrary)
    val classloaderForTestClass = this.getClass.getClassLoader
    println(classloaderForTestClass)
    this.getClass.getClassLoader.asInstanceOf[ScalaClassLoader.URLClassLoader]
  }
}

The output is:

scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@71c8becc
scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@71c8becc
java.lang.ClassCastException: scala.reflect.internal.util.ScalaClassLoader$URLClassLoader cannot be cast to scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
        at Main$.main(Test.scala:8)
        at Main.main(Test.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:98)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:32)
...

Why can't I cast ScalaClassLoader$URLClassLoader to ScalaClassLoader$URLClassLoader?

Edit:

On running:

scala -J-verbose:class Test.scala | grep ScalaClassLoader

The output is:

[Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from file:/C:/Development/Software/scala-2.12.2/lib/scala-reflect.jar]
...
...
[Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from file:/C:/DEVELO~1/Software/SCALA-~1.2/lib/scala-reflect.jar]

So there is definitely some shady class loading going on. Now trying to investigate why this is so

Weird, my classloader(`this.getClass.getClassLoader`) is `sun.misc.Launcher$AppClassLoader@68de145` and I can assert as instance of `URLClassLoader` (`this.getClass.getClassLoader.asInstanceOf[URLClassLoader]`). What about `new ScalaClassLoader.URLClassLoader(Seq(), null).asInstanceOf[ScalaClassLoader.URLClassLoader]`?? — prayagupa, May 25 '17 at 19:52
How are you starting your program? When I use `scala Test.scala` it works for me, when I run it from `sbt` or `IntelliJ` I get a `ClassCastException` with a different class loader. — Harald Gliebe, May 25 '17 at 19:52
@prayagupd You're probably running from an IDE which uses a different classloader. Try running from commandline — Kshitiz Sharma, May 25 '17 at 19:52
You are right. I was using IDE and class loader was Sun one. `scalac` gives me proper result and casting is working fine. `scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@e6ea0c6`. I'm using scala `2.12.2` — prayagupa, May 25 '17 at 19:56
`ClassCastExceptions` for the **same** class can happen when the class was loaded twice by different classloaders. In that case the classes are not considered equal by the JVM even though the have the same name. — Harald Gliebe, May 25 '17 at 19:57
@Harald As you can see in the screenshot. I don't have any customised classloaders — Kshitiz Sharma, May 25 '17 at 20:00
@Harald should `scala.reflect.internal.util.ScalaClassLoader$URLClassLoader` not be loaded only once. Curios who can load it twice? — prayagupa, May 25 '17 at 20:00
@prayagupd This is really weird. I though maybe something wrong with my scala installation, so just downloaded and tried 2.11.11. Same results — Kshitiz Sharma, May 25 '17 at 20:06
I think Herald's theory is right. https://stackoverflow.com/a/2371991/432903 — prayagupa, May 25 '17 at 20:07
@prayagupd Yes, normally it should be loaded only once, especially in such a simple case. But I've seen CCE in containers with dynamic classloaders where the same class could be loaded by unrelated classloaders. — Harald Gliebe, May 25 '17 at 20:08
@prayagupd That sounds reasonable. But the classloader seems to be the same. See my edit to question — Kshitiz Sharma, May 25 '17 at 20:11
@KshitizSharma: AFAICT the two classloaders have identical names, but _the classloaders in question_ (that are also Java classes) were in their turn _loaded by different classloaders_, likely from different jars. So, despite being "the same class", they cannot be cast to each other, because they potentially come from different source code, just named identically. — 9000, May 25 '17 at 20:14
@9000 The two classloaders seem to have same hashcode (see output). Which makes me think its the same instance. Assuming what you say is true, do you think there is a way to debug this further? — Kshitiz Sharma, May 25 '17 at 20:16
@KshitizSharma should `this.getClass.getClassLoader.getParent` tell who is the root `ClassLoader`? To me on `scalac` gives `null` which means itself is a Parent Loader. But IDE gives me `sun.misc.Launcher$ExtClassLoader@5034c75a`. — prayagupa, May 25 '17 at 20:29
Also, do `scala Test.scala -J-verbose:class | grep ScalaClassLoader` which gives all the `ScalaClassLoader` being loaded. for me there is one loading only `[Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from /usr/local/scala-2.12.2/lib/scala-reflect.jar` — prayagupa, May 25 '17 at 20:34
@prayagupd Thanks for pointing out the `-J-verbose:class` flag. It did reveal the double class loading. Now I need to figure out why — Kshitiz Sharma, May 25 '17 at 21:05
Sorry to see you're on Windows. Managing paths & classpaths on Windows is always challenging. Why does anyone use Windows? I need a corporate vpn client available only on Windows. — som-snytt, May 25 '17 at 21:51
@som-snytt Windows 10 has much better hardware and app support. Its command line is crap (even powershell), but that is solved by Cygwin. — Kshitiz Sharma, May 25 '17 at 22:29
@KshitizSharma :) I just updated cygwin and my environment broke. Of course it's not easy. :) Someday I'll try the Windows support for bash, on a day when I am less lazy. — som-snytt, May 26 '17 at 00:31

SergGr · Accepted Answer · 2017-05-25T22:38:43.980

If you extend your code a bit more as following:

import scala.reflect.internal.util.ScalaClassLoader

object test {

  def main(args: Array[String]) {
    val cl1 = this.getClass.getClassLoader
    println(cl1)
    val c1 = cl1.getClass 
    println(cl1.getClass)
    println(cl1.getClass.getClassLoader)

    println("-------")

    var c2 = classOf[ScalaClassLoader.URLClassLoader]
    println(c2)
    println(c2.getClassLoader)
    println("-------")
    println(c1 == c2)

  }
}

you'll get following output:

scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5cee5251
class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
sun.misc.Launcher$AppClassLoader@4554617c
-------
class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5cee5251
-------
false

Notice matching hashes @5cee5251. It means that first Scala interpreter loads ScalaClassLoader$URLClassLoader using root Java class loader and then uses that class loader to load all the classes in your script and when you ask for ScalaClassLoader$URLClassLoader in your code it is loaded with another (already loaded) instance of ScalaClassLoader$URLClassLoader. In this way your script is isolated from the "runtime environment" that executes it.

You may find some details at ScalaClassLoader.asContext method, that you can see in your stack trace, that uses Thread.setContextClassLoader to set itself as the main classloader for the thread that executes your script.

Update (Why it works on Mac but doesn't work on Windows)

The major difference between scala shell script for *nix and scala.bat for Windows is that by default on *nix platforms standard Scala libraries are added to Boot Classpath (see usebootcp in the script) while on Windows they are added to the "Usual Classpath". This is important because it defines which class loader will load scala.reflect.internal.util.ScalaClassLoader that is used by scala.tools.nsc.MainGenericRunner: will it be the root class loader (which is represented as null if you call getClassLoader) or Application Class Loader (i.e. an instance of sun.misc.Launcher$AppClassLoader). This is important because CommonRunner.run creates an instance of ScalaClassLoader using just urls without parent

def run(urls: Seq[URL], objectName: String, arguments: Seq[String]) {
  (ScalaClassLoader fromURLs urls).run(objectName, arguments)
}

This means that the parent class loader for the "main" ScalaClassLoader will be boot class loader rather than sun.misc.Launcher$AppClassLoader and thus when you ask this "main" ScalaClassLoader for class scala.reflect.internal.util.ScalaClassLoader it can't find it among classes loaded by its class loaders chain and thus has to load it again. This is the reason why you have two different instances of ScalaClassLoader class in your script.

There are two obvious workarounds (and both are not so good):

Change CommonRunner.run in the Scala source to actually pass current context class loader as the parent to the new ScalaClassLoader (might be not that easy ☺)
Change scala.bat to use -Xbootclasspath/a: instead of -cp for %_TOOL_CLASSPATH%. However looking into the usebootcp in the *nix script I can see following comment:

# default to the boot classpath for speed, except on cygwin/mingw/msys because
# JLine on Windows requires a custom DLL to be loaded.
unset usebootcp
if [[ -z "$cygwin$mingw$msys" ]]; then
  usebootcp="true"
fi

So I suspect that if you want to use scala.bat for REPL, moving all Scala libs to the Boot Classpath might be a bad idea. If this is the case, you probably need to create a copy of scala.bat (such as scala_run_script.bat) change it and use it to run your Scala scripts leaving standard scala.bat for REPL.

Interestingly on Mac OS with Scala 2.12.2 I get the follwing output `scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5fdef03a` `class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader` `null` `-------` `class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader` `null` `-------` `true` i.e. no `sun.misc.Launcher$AppClassLoader` involved and the final comparison returns `true` — Harald Gliebe, May 25 '17 at 20:32
Just like Herald, `c1 == c2` is `true` for me as well on scala 2.12.2 on Mac OS (using scalac compiler) but IDE has `false`. — prayagupa, May 25 '17 at 20:40
@Harald, @prayagupd do you compile it with `scalac` and then run or run this script directly using `scala` as OP seems to do? — SergGr, May 25 '17 at 20:41
@Harald, I only have Scala 2.11 on the Mac I have access to but surprisingly I got the same ("good") results there. Moreover, original example with `asInstanceOf` seems to work OK on that Mac as well. Could you check it in your environment? — SergGr, May 25 '17 at 20:49
@Harald Could this be a bug with handling of windows paths? Has anyone managed to run this on Windows? — Kshitiz Sharma, May 25 '17 at 21:06
@KshitizSharma, please take a look at my update with more details on reasons — SergGr, May 25 '17 at 22:39

Scala Script: Explain this class cast error to ScalaClassLoader on Windows

1 Answers1