Copy Json Flat file from local to HDFS

Question

package com.Main;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;



public class Main {

    public static void main(String[] args) throws IOException  {

        //Source file in the local file system
        String localSrc = args[0];

        //Destination file in HDFS
        String dst = args[1];

        //Input stream for the file in local file system to be written to HDFS
        InputStream in = new BufferedInputStream(new FileInputStream(localSrc));

        //Get configimport org.apache.commons.configuration.Configuration;uration of Hadoop system
        Configuration conf = new Configuration();
        System.out.println("Connecting to -- "+conf.get("fs.defaultFS"));

        //Destination file in HDFS
        FileSystem fs = FileSystem.get(URI.create(dst), conf);
        OutputStream out = fs.create(new Path(dst)); 

        //Copy file from local to HDFS
        IOUtils.copyBytes(in, out, 4096, true);

        System.out.println(dst + " copied to HDFS");
    }

}

AM getting following error message "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0 at com.Main.Main.main(Main.java:22)"

I have Json file in my local , have to move that in HDFS Ex: {"Del":"Ef77xvP","time":1509073785106}, {"Del":"2YXsF7r","time":1509073795109}

how are you running this? Are you providing arguments to the JAR at runtime? — philantrovert, Oct 28 '17 at 15:41

score 0 · Answer 1 · answered Oct 28 '17 at 16:00

0

Specify command line arguments to your program. You code snippet expects first argument to be source and next arguments to be destination. For more details refer to What is "String args[]"? parameter in main method Java

answered Oct 28 '17 at 16:00

Yogi Devendra

711
1
4
18

Specify command line arguments to your program --> Am not getting this, Can you please explain briefly. – user3214361 Oct 28 '17 at 16:47
eclipse: https://stackoverflow.com/questions/12222153/eclipse-how-we-take-arguments-for-main-when-run intellij: https://stackoverflow.com/questions/32951846/run-programme-from-intellij-with-command-line-file-input – Yogi Devendra Oct 29 '17 at 04:31

Copy Json Flat file from local to HDFS

1 Answers1