6

How to check whether a file in HDFS location is exist or not, using Oozie?

In my HDFS location I will get a file like this test_08_01_2016.csv at 11PM , on a daily basis.

I want check whether this file exist after 11.15 PM. I can schedule the batch using a Oozie coordinator job.

But how can I validate if the file exists in HDFS?

Alex Raj Kaliamoorthy
  • 2,035
  • 3
  • 29
  • 46
Sai
  • 1,075
  • 5
  • 31
  • 58

1 Answers1

6

you can use EL expression in oozie like:

<decision name="CheckFile">
         <switch>
            <case to="nextOozieTask">
              ${fs:exists('/path/test_08_01_2016.csv')} <!--do note the path which should be in ''-->
            </case>
            <default to="MailActionFileMissing" />
         </switch>
</decision>

You can also build the name of the file using simple shell script using capture output.

Alex Raj Kaliamoorthy
  • 2,035
  • 3
  • 29
  • 46
abhiieor
  • 3,132
  • 4
  • 30
  • 47
  • Hi Abhiieor, Thanks for ur reply.its working as expected :) ,i want to get today's date value dynamically to the file name i.e test_08_22_2016. i have tried coordination job to get the date value but it is not working..can please suggest on how can i get and append the date value in file. – Sai Aug 22 '16 at 11:24
  • Use http://stackoverflow.com/questions/1401482/yyyy-mm-dd-format-date-in-shell-script, convert - to _ (say date1) and then `export path='/path/test_'$date1'.csv'` – abhiieor Aug 22 '16 at 11:29
  • thanks ,i have tried above date format it will work as expected for todays date .but in my case if todays file is not there means i.e (test_08_22_2016) , i have to verify last one week files (test_08_21_2016,test_08_20_2016...etc). in that case how can i use this – Sai Aug 22 '16 at 11:48
  • generate date for last one week and create file names and check in oozie decision. – abhiieor Aug 22 '16 at 12:06
  • sorry ,can you please elaborate answer. i have tried to generate date from oozie coordination job, i am able to get date value from the coordination job but i am not able to add/append date value in file name in oozie job. – Sai Aug 22 '16 at 12:15