Hadoop with mongoDB : NoClassDefFoundError MongoConfigUtil

Question

I'm learning how to write a map / reduce job in hadoop with mongodb data as input. So I followed this example, but I got following error :

 Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/util/MongoConfigUtil
    at WordCount.main(WordCount.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.util.MongoConfigUtil
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

I've been searching for hours without any result. Any one can help me please.

have you tried -libjars options ? see this as an example https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html — Ram Ghadiyaram, May 18 '16 at 05:59
pom.xml is not visible. its very difficult to understand and answer what is missing how you are deploying — Ram Ghadiyaram, May 18 '16 at 06:01

Saravana · Answer 1 · 2016-05-18T08:00:28.717

1

it means mongo-hadoop-core.jar is available at compile time but not at run time.

Try running your application with dependency jars added in classpath

Example : java -cp mongo-hadoop-core.jar<++other dependencies> MainClass

EDIT 1

If you're running using hadoop shell

check the classpath by executing hadoop classpath it will print the dir/jars in the classpath.

If the dependent jars are not in the classpath add them in the classpath using export command then execute hadoop jar yourjar.jar mainClass

EDIT 2

make use of libjars option

hadoop jar myjar.jar mainClass -libjars ${LIBJARS}

edited May 18 '16 at 08:00

answered May 18 '16 at 03:47

Saravana

12,647
2
39
57

The problem is that in hadoop we should extract the jar from the application and execute it with "hadoop jar name.jar className" so i don't know how to include other jars with it. – Namsi Abdelkhalek May 18 '16 at 05:03

score 0 · Answer 2 · edited Oct 27 '17 at 06:41

I can see this link examples folder structure is maven. I CANT see pom.xml in that link.

We can set maven scope from provided to runtime

You need to write assembly.xml (to package your application lib and related dependencies in tar file) and need to refer it from pom.xml to package mongo-hadoop-core.jar along with other dependencies which are NOT installed in cluster.

export HADOOP_CLASSPATH=`hadoop classpath`:`hbase classpath`

hadoop jar .... -cp $HADOOP_CLASSPATH MainClass

If you unzip the tar file mentioned above and you can refer the classpath for example : hadoop jar .... ../lib/* main classs where lib folder contails all your dependencies like your mongodb also.

If mongo db and related components are installed in the cluster we can mention the classpath like below example.

Also see this answer how they have used libjars

Hadoop with mongoDB : NoClassDefFoundError MongoConfigUtil

2 Answers2