Getting NullPointerException while running a normal Mapreduce program in Eclipse

Question

I am getting NullPointerException when trying to execute a simple MapReduce program. I am unable to understand where the problem is?

package MapReduce.HadMapReduce;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class RecCount extends Configured implements Tool {

    public int run(String[] arg0) throws Exception {

        Job job = Job.getInstance(getConf());

        FileInputFormat.setInputPaths(job, new Path("C:\\singledeck.txt"));
        FileOutputFormat.setOutputPath(job, new Path("C:\\temp123"));

        return job.waitForCompletion(true) ? 0 : 1;
    }

    public static void main(String args[]) throws Exception {
        System.exit(ToolRunner.run(new RecCount(), args));
    }
}

Error is:

Exception in thread "main" java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:483)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:731)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:489)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:530)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:507)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
    at MapReduce.HadMapReduce.RecCount.run(RecCount.java:22)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at MapReduce.HadMapReduce.RecCount.main(RecCount.java:26)

This is the logic which is happening behind the scenes:

ToolRunner is calling the below run method and this method calls its other run method (which is pasted right below this) where the configuration is set if it is null.

public static int run(Tool tool, String[] args) throws Exception {
    return run(tool.getConf(), tool, args);
}        

public static int run(Configuration conf, Tool tool, String[] args) throws  Exception {
    if (conf == null) {
        conf = new Configuration();
    }
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    // set the configuration back, so that Tool can configure itself
    tool.setConf(conf);

    // get the args w/o generic hadoop args
    String[] toolArgs = parser.getRemainingArgs();
    return tool.run(toolArgs);
}

In the last statement above run method is called which I implemented because I implemented Tool interface. I don't see any error in my code. Please let me know if you can find any!

Can someone please explain what is the problem with my code?

I *think* the problem is that the code is not allowed to write into the C: root folder. Please try with a "harmless" path, like create a C:\temp folder read-write accessible for anyone, and use that for the in and output... I think this because the problem arises in the "setPermission". But this is really just a wild stab in the dark. — ppeterka, Aug 14 '16 at 20:17
Possible duplicate of [What is a NullPointerException, and how do I fix it?](http://stackoverflow.com/questions/218384/what-is-a-nullpointerexception-and-how-do-i-fix-it) — hotzst, Aug 14 '16 at 20:18
@hotzst The NPE is in a 3rd party library outside of the hands of OP - that might be a different case... — ppeterka, Aug 14 '16 at 20:18
@kolboc : No i did not provide any args i just ran it through an ide after i imported all the need hadoop jar files — Valar Morghulis, Aug 14 '16 at 20:54
@ppeterka : I tried creating a new directory under C and copied my input file in that. But still am getting the same error!!!!! — Valar Morghulis, Aug 14 '16 at 20:58
@ppeterka : Please see my edits in the question, i gave a proper explanation of what is happening behind the scenes of the run method. the configuration is set but still am getting error. Please let me know what is the problem here. If possible please write a piece of code and explain? — Valar Morghulis, Aug 14 '16 at 22:05
Possible duplicate of [Is it possible to run Hadoop jobs (like the WordCount sample) in the local mode on Windows without Cygwin?](http://stackoverflow.com/questions/26516865/is-it-possible-to-run-hadoop-jobs-like-the-wordcount-sample-in-the-local-mode) — OneCricketeer, Aug 14 '16 at 22:18

Getting NullPointerException while running a normal Mapreduce program in Eclipse

0 Answers0