pyspark: How to submit depend files & folders?

Question

I want to submit a pyspark task. And some .py files in different folders.Especially I want put configuration files and common tools in only one folder. But when I submit a pyspark task, I just know --py-files param, so how to submit folders? My code struct likes:

--conf folder
|  --origin.conf
|  --scenes.conf
--tools folder
|  --utils.py
|  --vali.py
-- other fodlsers...

@Yaron For example, a file --plot_distribution.py-- in folder data_analysis ,has a statement --from tools import utils--,which tools is also a folder. And other py files also depend on utils.py,so I want to submit folders not only files. — Peng He, Nov 13 '16 at 14:26

score 4 · Accepted Answer · answered Nov 14 '16 at 04:49

4

create Python package to organize the code
zip package or create egg file
submit your app passing egg or zip file to --py-files / sc.pyFiles

answered Nov 14 '16 at 04:49

score 0 · Answer 2 · answered Dec 01 '17 at 00:16

0

This link from Cloudera has some examples of distributing python packages to Spark executors Running Spark Python Applications

answered Dec 01 '17 at 00:16

WhyNot

47
4

pyspark: How to submit depend files & folders?

2 Answers2