8

For a project, I'm using spark structured streaming with kafka.

I have this conf :

    <spark.version>3.1.1</spark.version>
    <slf4j.version>1.7.7</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <scala.version>2.12</scala.version>

But when I run, I have this error :

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0

I'm confused because I alrady import databind in 2.12.3. Just here, an extract of my pom.xml :

    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-yaml</artifactId>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_${scala.version}</artifactId>
        <version>2.12.3</version>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.12.3</version>
    </dependency>

thank you in advance for helping

Mathieu
  • 81
  • 1
  • 1
  • 3

2 Answers2

0

Are you sure of the error? Scala 2.12 requires Jackson 2.10.x.

To debug, run mvn dependency:tree to see what dependencies/versions are actually being dragged in. You are probably getting other Jackson JARs that are not 2.10.x.

If you are building a Spring project (with parent POM spring-boot-dependencies), set:

<jackson-bom.version>2.10.5.20201202</jackson-bom.version>

for the latest Jackson that satisfies the Scala requirement. Otherwise, you'll need to add the following to your <dependencyManagement/>:

  <dependencyManagement>                                                        
    <dependencies>
      ...
      <dependency>                                                              
        <groupId>com.fasterxml.jackson</groupId>                                
        <artifactId>jackson-bom</artifactId>                                    
        <version>${jackson-bom.version}</version>                               
        <type>pom</type>                                                        
        <scope>import</scope>                                                   
      </dependency>
      ...
    </dependencies>                                                             
  </dependencyManagement>

and remove the <version> from you Jackson dependencies.

Allen D. Ball
  • 1,916
  • 1
  • 9
  • 16
  • Thanks for your answer. When I try your exact proposition, I have this error : Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 requires Jackson Databind version >= 2.10.0 and < 2.11.0 I also tried to do : 2.12.3 In this case, I have the same error as the start. So : Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0 I don't undrestant because jackson databind is in 2.12.3 – Mathieu Jun 01 '21 at 07:33
  • Please add your whole pom.xml to the question and the output of `mvn dependency:tree`. Ideally, also please add a cut-and-paste of the exception you're getting. – Allen D. Ball Jun 01 '21 at 14:21
  • Hi, I respond to you in comments because it's too long. Thanks – Mathieu Jun 07 '21 at 14:52
0

Allen D.Ball, my answer is too long for comment. So, pom.xml :

4.0.0

<parent>
    <groupId>fr.cnp.kfk</groupId>
    <artifactId>jee</artifactId>
    <version>0.0.1-SNAPSHOT</version>
</parent>

<artifactId>kfk-spark-structured-streams</artifactId>
<packaging>jar</packaging>

<name>Spark Structured Streams</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <spark.version>3.1.1</spark.version>
    <slf4j.version>1.7.7</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <scala.version>2.12</scala.version>
    <jackson-bom.version>2.12.3</jackson-bom.version>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <artifactSet>
                            <excludes>
                                <exclude>org.apache.flink:force-shading</exclude>
                                <exclude>com.google.code.findbugs:jsr305</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>fr.cnp.kfk.streams.Application</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>

    <pluginManagement>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>org.eclipse.m2e</groupId>
                <artifactId>lifecycle-mapping</artifactId>
                <version>1.0.0</version>
                <configuration>
                    <lifecycleMappingMetadata>
                        <pluginExecutions>
                            <pluginExecution>
                                <pluginExecutionFilter>
                                    <groupId>org.apache.maven.plugins</groupId>
                                    <artifactId>maven-assembly-plugin</artifactId>
                                    <versionRange>[2.4,)</versionRange>
                                    <goals>
                                        <goal>single</goal>
                                    </goals>
                                </pluginExecutionFilter>
                                <action>
                                    <ignore />
                                </action>
                            </pluginExecution>
                            <pluginExecution>
                                <pluginExecutionFilter>
                                    <groupId>org.apache.maven.plugins</groupId>
                                    <artifactId>maven-compiler-plugin</artifactId>
                                    <versionRange>[3.1,)</versionRange>
                                    <goals>
                                        <goal>testCompile</goal>
                                        <goal>compile</goal>
                                    </goals>
                                </pluginExecutionFilter>
                                <action>
                                    <ignore />
                                </action>
                            </pluginExecution>
                        </pluginExecutions>
                    </lifecycleMappingMetadata>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

<dependencies>
    <dependency>
        <groupId>fr.cnp.kfk</groupId>
        <artifactId>kfk-commons</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.version}</artifactId>
        <version>${spark.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.codehaus.janino</groupId>
                <artifactId>janino</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.codehaus.janino</groupId>
                <artifactId>commons-compiler</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
        <version>3.1.1</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.commons</groupId>
                <artifactId>commons-pool2</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>

    <dependency>
        <groupId>com.opencsv</groupId>
        <artifactId>opencsv</artifactId>
        <version>4.1</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-yaml</artifactId>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_${scala.version}</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
    </dependency>

    <!-- Dépendences pour RecksDB -->
    <!-- <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb_2.12</artifactId> 
        <version>${flink.version}</version> <scope>provided</scope> </dependency> -->

    <dependency>
        <artifactId>janino</artifactId>
        <groupId>org.codehaus.janino</groupId>
        <version>3.0.8</version>
    </dependency>
    <dependency>
        <artifactId>commons-compiler</artifactId>
        <groupId>org.codehaus.janino</groupId>
        <version>3.0.8</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-pool2</artifactId>
        <version>2.9.0</version>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson</groupId>
            <artifactId>jackson-bom</artifactId>
            <version>${jackson-bom.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

dependency.tree :

[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ kfk-spark-structured-streams --- 
[INFO] fr.cnp.kfk:kfk-spark-structured-streams:jar:0.0.1-SNAPSHOT 
[INFO] +- fr.cnp.kfk:kfk-commons:jar:0.0.1-SNAPSHOT:compile 
[INFO] +- org.apache.spark:spark-sql_2.12:jar:3.1.1:compile 
[INFO] |  +- com.univocity:univocity-parsers:jar:2.9.1:compile 
[INFO] |  +- org.apache.spark:spark-sketch_2.12:jar:3.1.1:compile 
[INFO] |  +- org.apache.spark:spark-core_2.12:jar:3.1.1:compile 
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile 
[INFO] |  |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile 
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile 
[INFO] |  |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile 
[INFO] |  |  |  \- org.apache.avro:avro-ipc:jar:1.8.2:compile 
[INFO] |  |  +- com.twitter:chill_2.12:jar:0.9.5:compile 
[INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile 
[INFO] |  |  |        +- com.esotericsoftware:minlog:jar:1.3.0:compile 
[INFO] |  |  |     \- org.objenesis:objenesis:jar:2.6:compile 
[INFO] |  |  +- com.twitter:chill-java:jar:0.9.5:compile 
[INFO] |  |  +- org.apache.hadoop:hadoop-client:jar:3.2.0:compile 
[INFO] |  |  |  +- org.apache.hadoop:hadoop-common:jar:3.2.0:compile 
[INFO] |  |  |  |     +- com.google.guava:guava:jar:23.0:compile 
[INFO] |  |  |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.0.18:compile
[INFO] |  |  |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile 
[INFO] |  |  |  | |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile [INFO] |  |  |  |  +- commons-cli:commons-cli:jar:1.3.1:compile 
[INFO] |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.5:compile 
[INFO] |  |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.9:compile 
[INFO] |  |  |  |    +- commons-io:commons-io:jar:2.5:compile 
[INFO] |  |  |  |  +- org.eclipse.jetty:jetty-servlet:jar:9.4.8.v20171121:compile 
[INFO] |  |  |  |  +- javax.servlet.jsp:jsp-api:jar:2.2:provided 
[INFO] |  |  |  |  +- org.apache.commons:commons-configuration2:jar:2.1.1:compile 
[INFO] |  |  |  |  +- com.google.re2j:re2j:jar:1.1:compile 
[INFO] |  | |  |  +- com.google.code.gson:gson:jar:2.8.2:compile 
[INFO] |  |  |  |    +- org.apache.hadoop:hadoop-auth:jar:3.2.0:compile 
[INFO] |  |  |  |  |  +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile 
[INFO] |  |  |  |  |  |  \- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile [INFO] |  |  |  |  |  \- net.minidev:json-smart:jar:2.3:compile 
[INFO] |  |  |  |  |     \- net.minidev:accessors-smart:jar:1.2:compile 
[INFO] |  |  | |  +- org.apache.curator:curator-client:jar:2.12.0:compile 
[INFO] |  | |  |  +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:compile 
[INFO] |  |  |  |  +- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile 
[INFO] |  |  |  |  |    +- org.apache.kerby:kerb-client:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  +- org.apache.kerby:kerby-config:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  +- org.apache.kerby:kerb-core:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  |  \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  |     +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  |     \- org.apache.kerby:kerby-util:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  +- org.apache.kerby:kerb-common:jar:1.0.1:compile 
[INFO] |  | |  |  |  |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  +- org.apache.kerby:kerb-util:jar:1.0.1:compile 
[INFO] |  |  |  |  |  |  \- org.apache.kerby:token-provider:jar:1.0.1:compile 
[INFO] |  |  |  | |  \- org.apache.kerby:kerb-admin:jar:1.0.1:compile 
[INFO] |  |  |  |  |     +- org.apache.kerby:kerb-server:jar:1.0.1:compile 
[INFO] |  |  | |  |     |  \- org.apache.kerby:kerb-identity:jar:1.0.1:compile 
[INFO] |  |  |  |  |     \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile 
[INFO] |  |  |  |  +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile 
[INFO] |  |  |  |     +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile 
[INFO] |  |  |  |  \- dnsjava:dnsjava:jar:2.1.7:compile 
[INFO] |  |  |  +- org.apache.hadoop:hadoop-hdfs-client:jar:3.2.0:compile 
[INFO] |  |  |  |  \- com.squareup.okhttp:okhttp:jar:2.7.5:compile 
[INFO] |  |  |  |   \- com.squareup.okio:okio:jar:1.6.0:compile 
[INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:3.2.0:compile 
[INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.3.0:provided 
[INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:3.2.0:compile 
[INFO] |  |  |     +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.2.0:compile 
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:3.2.0:compile 
[INFO] |  |  |  |     +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.3:compile 
[INFO] |  |  |  |     |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile 
[INFO] |  |  | |     |  \- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile [INFO] |  |  |  |     \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.3:compile 
[INFO] |  |  |  |        \- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.3:compile 
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.2.0:compile [INFO] |  |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.2.0:compile [INFO] |  |  |  \- org.apache.hadoop:hadoop-annotations:jar:3.2.0:compile 
[INFO] |  |  +- org.apache.spark:spark-launcher_2.12:jar:3.1.1:compile 
[INFO] |  |  +- org.apache.spark:spark-kvstore_2.12:jar:3.1.1:compile 
[INFO] |  |  |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile 
[INFO] |  |  +- org.apache.spark:spark-network-common_2.12:jar:3.1.1:compile 
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.12:jar:3.1.1:compile 
[INFO] | |  +- org.apache.spark:spark-unsafe_2.12:jar:3.1.1:compile 
[INFO] |  |    +- javax.activation:activation:jar:1.1.1:provided 
[INFO] |  |  +- org.apache.curator:curator-recipes:jar:2.13.0:compile 
[INFO] |  |  |  \- org.apache.curator:curator-framework:jar:2.13.0:compile 
[INFO] |  |    +- org.apache.zookeeper:zookeeper:jar:3.4.14:compile 
[INFO] |  |  |  \- org.apache.yetus:audience-annotations:jar:0.5.0:compile 
[INFO] |  |    +- jakarta.servlet:jakarta.servlet-api:jar:4.0.3:compile 
[INFO] |  |  +- org.apache.commons:commons-math3:jar:3.4.1:compile 
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.0:compile 
[INFO] |  |  +- org.slf4j:slf4j-api:jar:1.7.25:compile 
[INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.25:compile 
[INFO] |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.25:compile 
[INFO] |  |  +- log4j:log4j:jar:1.2.17:compile 
[INFO] |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile 
[INFO] |  |  +- com.ning:compress-lzf:jar:1.0.3:compile 
[INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile 
[INFO] |  |  +- org.lz4:lz4-java:jar:1.7.1:compile 
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.4.8-1:compile 
[INFO] |  |  +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:compile 
[INFO] |  |  |  \- org.roaringbitmap:shims:jar:0.9.0:runtime 
[INFO] |  |  +- commons-net:commons-net:jar:3.1:compile 
[INFO] |  |  +- org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:compile 
[INFO] |  |     +- org.scala-lang:scala-reflect:jar:2.12.10:compile 
[INFO] |  |  +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M5:compile 
[INFO] |  |  |  \- org.json4s:json4s-core_2.12:jar:3.7.0-M5:compile 
[INFO] |  |  |     +- org.json4s:json4s-ast_2.12:jar:3.7.0-M5:compile 
[INFO] |  |  |     \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M5:compile 
[INFO] |  |  +- org.glassfish.jersey.core:jersey-client:jar:2.25.1:compile 
[INFO] |  | |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:provided 
[INFO] |  |  |  +- org.glassfish.hk2:hk2-api:jar:2.5.0-b32:compile 
[INFO] |  |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b32:compile 
[INFO] |  |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b32:compile 
[INFO] |  |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b32:compile 
[INFO] | |  |  \- org.glassfish.hk2:hk2-locator:jar:2.5.0-b32:compile 
[INFO] |  |  +- org.glassfish.jersey.core:jersey-common:jar:2.25.1:compile 
[INFO] |  |  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile 
[INFO] |  |  |     +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.25.1:compile 
[INFO] |  |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile 
[INFO] |  |     +- org.glassfish.jersey.core:jersey-server:jar:2.25.1:compile
[INFO] |  |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.25.1:compile 
[INFO] |  |  |  \- javax.validation:validation-api:jar:1.1.0.Final:compile 
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.25.1:compile 
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:compile 
[INFO] |  |  +- org.glassfish.jersey.inject:jersey-hk2:jar:2.30:compile 
[INFO] |  |  | \- org.javassist:javassist:jar:3.21.0-GA:compile 
[INFO] |  |  +- io.netty:netty-all:jar:4.1.51.Final:compile 
[INFO] |  |  +- com.clearspring.analytics:stream:jar:2.9.6:compile 
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.1.5:compile 
[INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.5:compile 
[INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.1.5:compile 
[INFO] |  |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.5:compile 
[INFO] |  |     +- io.dropwizard.metrics:metrics-jmx:jar:4.1.1:compile 
[INFO] |  |  +- org.apache.ivy:ivy:jar:2.4.0:compile 
[INFO] |  |  +- oro:oro:jar:2.0.8:compile 
[INFO] |  |  +- net.razorvine:pyrolite:jar:4.30:compile 
[INFO] |  |  +- net.sf.py4j:py4j:jar:0.10.9:compile 
[INFO] |  |  \- org.apache.commons:commons-crypto:jar:1.1.0:compile 
[INFO] |  +- org.apache.spark:spark-catalyst_2.12:jar:3.1.1:compile 
[INFO] |  |  +- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:compile [INFO] |  |  +- org.antlr:antlr4-runtime:jar:4.8-1:compile 
[INFO] |  |    +- commons-codec:commons-codec:jar:1.10:compile 
[INFO] |  |  +- org.apache.arrow:arrow-vector:jar:2.0.0:compile 
[INFO] |  |  |  +- org.apache.arrow:arrow-format:jar:2.0.0:compile 
[INFO] |  |  |  +- org.apache.arrow:arrow-memory-core:jar:2.0.0:compile 
[INFO] |  |  |  \- com.google.flatbuffers:flatbuffers-java:jar:1.9.0:compile 
[INFO] |  |  \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:compile 
[INFO] |     +- org.apache.spark:spark-tags_2.12:jar:3.1.1:compile 
[INFO] |  +- org.apache.orc:orc-core:jar:1.5.12:compile 
[INFO] |  |  +- org.apache.orc:orc-shims:jar:1.5.12:compile 
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile 
[INFO] |  |  +- commons-lang:commons-lang:jar:2.6:compile 
[INFO] |  |  +- io.airlift:aircompressor:jar:0.10:compile 
[INFO] |  |  \- org.threeten:threeten-extra:jar:1.5.0:compile 
[INFO] |  +- org.apache.orc:orc-mapreduce:jar:1.5.12:compile 
[INFO] |  +- org.apache.hive:hive-storage-api:jar:2.7.2:compile 
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.10.1:compile 
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.10.1:compile 
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.10.1:compile 
[INFO] |  +- org.apache.parquet:parquet-hadoop:jar:1.10.1:compile 
[INFO] |  |  +- org.apache.parquet:parquet-format:jar:2.4.0:compile 
[INFO] |  |  +- org.apache.parquet:parquet-jackson:jar:1.10.1:compile 
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile 
[INFO] |  | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile 
[INFO] |     +- org.apache.xbean:xbean-asm7-shaded:jar:4.15:compile 
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile 
[INFO] +- org.apache.spark:spark-sql-kafka-0-10_2.12:jar:3.1.1:compile 
[INFO] |     +- org.apache.spark:spark-token-provider-kafka-0-10_2.12:jar:3.1.1:compile 
[INFO] |  \- org.apache.kafka:kafka-clients:jar:2.6.0:compile 
[INFO]    +- org.apache.spark:spark-streaming_2.12:jar:3.1.1:compile 
[INFO] |  \- org.scala-lang:scala-library:jar:2.12.10:compile 
[INFO] +- com.opencsv:opencsv:jar:4.1:compile 
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:compile 
[INFO] |  +- org.apache.commons:commons-text:jar:1.1:compile 
[INFO] |  \- commons-beanutils:commons-beanutils:jar:1.9.3:compile 
[INFO] |     \- commons-collections:commons-collections:jar:3.2.2:compile 
[INFO] +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.12.3:compile [INFO] |  +- org.yaml:snakeyaml:jar:1.17:compile 
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile 
[INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.12.3:runtime 
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.3:compile 
[INFO] |  \- com.thoughtworks.paranamer:paranamer:jar:2.8:compile 
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.3:compile 
[INFO]    +- org.codehaus.janino:janino:jar:3.0.8:compile 
[INFO] +- org.codehaus.janino:commons-compiler:jar:3.0.8:compile 
[INFO] \- org.apache.commons:commons-pool2:jar:2.9.0:compile 
[INFO]    ------------------------------------------------------------------------ 
[INFO] BUILD SUCCESS 
[INFO]    ------------------------------------------------------------------------ 
[INFO] Total time:  7.334 s [INFO] Finished at: 2021-06-02T09:19:26+02:00 
[INFO]    ------------------------------------------------------------------------

The error :

Error: Bad target:null
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at fr.cnp.kfk.streams.Application.main(Application.java:14)
Caused by: org.apache.spark.sql.streaming.StreamingQueryException: null

[...]

    at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:354)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
    at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:338)
    at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2$(WriteToDataSourceV2Exec.scala:336)
    at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.writeWithV2(WriteToDataSourceV2Exec.scala:297)
    at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.run(WriteToDataSourceV2Exec.scala:304)
    at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)
    at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)
    at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)
    at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3696)
    at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2965)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
    at org.apache.spark.sql.Dataset.collect(Dataset.scala:2965)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:589)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
    at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
    at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
    at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
    at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
    at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
    at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
    at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
    ... 1 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 requires Jackson Databind version >= 2.10.0 and < 2.11.0
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
    at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:747)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    ... 40 more
Mathieu
  • 81
  • 1
  • 1
  • 3