1

When I submit my python file to spark like this 

spark-submit driver.py

It starts showing a lot of warning related to python 2 print method.

18/10/19 01:37:52 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.5.1.112
ExitCodeException exitCode=1:   File "/etc/hadoop/conf/topology_script.py", line 63
    print rack
             ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?

Is there any way like spark-submit driver.py -W ignore ??

>> I know this is the warning of python2, my code is in python3 so I just want to ignore the python2 warnings. These warnings start showing even before control goes to my code. Because spark-submit runs first and later on load python files.

Avinash
  • 2,093
  • 4
  • 28
  • 41

1 Answers1

3

You're using python 3, but writing in Python 2?

Please refer to this question : What does "SyntaxError: Missing parentheses in call to 'print'" mean in Python?

You can also refer to Configure logging in Apache Spark to set the default loglevel using a log4j.properties file.

You can also set the loglevel when you call spark-submit or inside the code using sc.setLogLevel("WARN")

Just another SO question for reference : How to stop INFO messages displaying on spark console?

pissall
  • 7,109
  • 2
  • 25
  • 45
  • I am using python 3, Problem is `spark-submit` runs first and generates the warning. You can see it is complaining `/etc/hadoop/conf/topology_script.py` which is hadoop file, My code is `driver.py` – Avinash Oct 19 '18 at 06:40
  • @Avinash Please close the question if you have your answer – pissall Sep 20 '19 at 12:07