How to check Spark Version

Question

I want to check the spark version in cdh 5.7.0. I have searched on the internet but not able to understand. Please help.

See [When is it justifiable to downvote a question?](http://meta.stackoverflow.com/questions/252677/when-is-it-justifiable-to-downvote-a-question) on [meta]. — tripleee, Jul 27 '16 at 07:27
but i tried few things before posting it. I didn't find any help thats why i posted here — Ironman, Jul 27 '16 at 07:28
Programmatically finding spark version would be best. That's for sure the one that my code is using. How to do it in python? Thanx — Geoffrey Anderson, May 24 '18 at 16:25
#print(pyspark.__version()__) #print(pyspark.version()) import pyspark #print(pyspark.version()) #print(pyspark.__version()__) print(pyspark.__version__) # this works — Geoffrey Anderson, May 24 '18 at 16:28

score 125 · Accepted Answer · edited Sep 07 '18 at 18:27

125

Addition to @Binary Nerd

If you are using Spark, use the following to get the Spark version:

spark-submit --version

or

Login to the Cloudera Manager and goto Hosts page then run inspect hosts in cluster

edited Sep 07 '18 at 18:27

tk421

answered Jul 26 '16 at 10:22

BruceWayne

I get "ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom$ spark-submit --version >spark-submit: command not found – Geoffrey Anderson May 24 '18 at 16:26
Check your SPARK_HOME config – BruceWayne May 25 '18 at 03:18
I got a `NameError: name 'submit' is not defined` error – Gabriel Fair Dec 07 '18 at 00:30
can u share your command – BruceWayne Dec 10 '18 at 04:31
Too bad the output has ridiculous ASCII art that makes it difficult to parse :( – Jeff Evans Mar 29 '19 at 19:46

score 33 · Answer 2 · answered Jul 26 '16 at 11:05

33

You can get the spark version by using the following command:

spark-submit --version

spark-shell --version

spark-sql --version

You can visit the below site to know the spark-version used in CDH 5.7.0

answered Jul 26 '16 at 11:05

spark-sql --version is not working. Above two are working fine – Farooque Nov 15 '18 at 09:57

score 2 · Answer 3 · answered Jul 26 '16 at 10:16

2

According to the Cloudera documentation - What's New in CDH 5.7.0 it includes Spark 1.6.0.

answered Jul 26 '16 at 10:16

Binary Nerd

3 Answers3