0

My Storm class uses Redis queue to collect data.

I tried to run my Storm jar by 

storm jar jar_file_name.jar Topology_name configuration_file

But I got the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: Lredis/clients/jedis/Jedis;
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
    at java.lang.Class.getDeclaredField(Class.java:1946)
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at backtype.storm.utils.Utils.serialize(Utils.java:77)
    at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:111)
    at OutlierPredictor.main(OutlierPredictor.java:98)
 Caused by: java.lang.ClassNotFoundException: redis.clients.jedis.Jedis
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

I have compiled using

javac -classpath $HADOOP_CORE:$HBASE_CLASSPATH:/usr/local/hadoop-  2.2.0/redis_jar/commons-pool-1.5.5.jar:/usr/local/hadoop-2.2.0/redis_jar/jedis-2.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar -d dir_name/ dir_name/Javafile.java

from command line.

I am executing this in a single node set up.

What is going wrong?

Regent
  • 5,142
  • 3
  • 21
  • 35
Sushmita Bhattacharya
  • 453
  • 2
  • 7
  • 15
  • I suggest using a project management tool like maven to care about the cp, compiling and uberjar. – halfelf Sep 03 '14 at 06:45

5 Answers5

1

You need to package all your code and dependencies into a single jar.

Refer to the Storm tutorial, Topologies section,

Running a topology is straightforward. First, you package all your code and dependencies into a single jar. Then, you run a command like the following:

storm jar all-my-code.jar backtype.storm.MyTopology arg1 arg2

You may have to use some packaging tool like, OneJAR, JarJar or ANT and create a jar containing all your files and dependencies. Please refer to these SO posts

  1. Merging Multiple Jars in to a Single Jar
  2. Easiest way to merge a release into one JAR file
Community
  • 1
  • 1
Vishal John
  • 4,231
  • 25
  • 41
1

1)add jedis in dependency

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>2.9.0</version>
</dependency>

2)use assembly plugin to package all dependencies into one jar file

   <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.6</version>
        <configuration>
            <manifest>
                <addClasspath>true</addClasspath>
                <mainClass>your main class</mainClass>
            </manifest>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
            <archive>
                <manifest>
                    <mainClass>your main class</mainClass>
                </manifest>
            </archive>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
        <executions>
            <execution>
                <id>make-assembly</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

3) ignore storm-core in package

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-core</artifactId>
    <version>1.1.1</version>
    <scope>provided</scope>
</dependency>
skywalker
  • 11
  • 2
  • Please use the Maven dependency as said earlier or use the Jedis jar provided in the github repository – sathya Oct 17 '17 at 07:34
0

Thank you John for the answer .

But I found another solution.

I did not included jedis-2.1.0.jar, commons-pool-1.5.5.jar in my $STORM_HOME/lib/

Note: this two files were there in the $REDIS_CORE.

Sushmita Bhattacharya
  • 453
  • 2
  • 7
  • 15
0

JedisPoolConfig is needed when we use Jedis Configuration. In Spring Boot 2.0, spring-boot-starter-data-redis gives Lettuce dependency by default instead of Jedis. To use Jedis configuration, exclude Lettuce and add Jedis as following.

    <dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-redis</artifactId>
  <exclusions>
    <exclusion>
     <groupId>io.lettuce</groupId>
     <artifactId>lettuce-core</artifactId>
    </exclusion>
  </exclusions>            
</dependency>        
<dependency>
  <groupId>redis.clients</groupId>
  <artifactId>jedis</artifactId>
</dependency>
Sachin Sridhar
  • 434
  • 4
  • 13
0

information for the following issues regarding the higher version of Spring Data Redis:

Due to the amount of breaking changes in Jedis 4, Spring Data Redis 2.x cannot be used with Jedis 4.x. Jedis 4.x support will be part of Data Redis 3.0

Oskarro
  • 323
  • 2
  • 9