My code can run normally in idea by local mode, When I printed it into a jar package and uploaded it to the SPARK server I deployed to run, NoSuchMethodError: scala. Predef $. refArrayOps appeared.
The line of code that went wrong is as follows
val expectArray=expectVertex.take(2).toArray.sortBy（it=>｛it_1｝）
the expectVertex is a scala map,its key type is graphx.VertexId,its value type is Int
I have also encountered this issue when doing this with Spark simple code, This error occurred when I was using a row of an array function，the code like this package org.example
import org.apache.spark.graphx.{Edge, Graph}
import org.apache.spark.{SparkConf, SparkContext}
import java.util.logging.{Level, Logger}
/**
* Hello world!
*
*/
class App{
def run(): Unit ={
Logger.getLogger("org.apache.spark").setLevel(Level.WARNING)
Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF)
val conf = new SparkConf().setAppName("AXU test")
.setMaster("local")
val sc = new SparkContext(conf)
val vertices = sc.parallelize(Array((1L, "A"), (2L, "B"), (3L, "C"), (4L, "D")))
val edges = sc.parallelize(Array(Edge(1L, 2L, "friend"), Edge(2L, 3L, "follow"), Edge(3L, 4L, "friend")))
val graph = Graph(vertices, edges)
val inDegrees = graph.inDegrees
inDegrees.collect().foreach(println)
val deg = inDegrees.collect()
for( i <- 0 to deg.length-1){
print("this is no." + (i+1) + " point indegree:")
println("id: " + deg(i)._1 + " value: " + deg(i)._2)
}
sc.stop()
}
}
the log is
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
at org.example.App.run(App.scala:23)
at org.example.Main$.main(Main.scala:6)
at org.example.Main.main(Main.scala)
if i remove the code in line no.23, the code is
inDegrees.collect().foreach(println)
it can work normally.
My compiled and running versions of scala are both 2.12.7.
It looks like I can't use methods like Array [T]. foreach or Array [T]. sortBy (it=>{it_1}) in jar packages(I used Maven to package the jar).
the maven content is below.
<properties>
<scala.version>2.12.7</scala.version>
<spark.version>2.4.4</spark.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>org.example.Main</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>assembly</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>org.example.Main</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
Can someone tell me why appear this problem? Thank you in advance.
