0

I'm bit new to Spark and trying to understand few term. (Couldn't understand using online resources)

Please validate me first with below terms:

Executor: Its container or JVM process which will be running on worker node or data node. We can have multiple Executors per node.

Core: Its a thread within a container or JVM process running on worker node or data node. We can have multiple cores or threads per executor.

Please correct me If am wrong in above two concepts.

Questions:

  1. When ever we submit spark job, What does it means ? Are we handing over our job to Yarn or resource manager which will assigning resources to my application or job in cluster and execute that ? Its it correct understanding .. ?

  2. In command used to submit job in spark cluster, there is an option to set number of executors.

    spark-submit --class <CLASS_NAME> --num-executors ? --executor-cores ? --executor-memory ? ....

So these number of executors + cores will be setting up per-node? If not then how can we set specific number of cores per node?

QuickSilver
  • 3,915
  • 2
  • 13
  • 29
Maria
  • 452
  • 1
  • 6
  • 20

1 Answers1

0

All of your assumptions are correct. For in a detailed explanation regarding cluster architecture please go through this link. You'll get a clear picture. Regarding your second question, num-of-executors is for the entire cluster. It is calculated as below: 

num-cores-per-node * total-nodes-in-cluster

For example, suppose that you have a 20-node cluster with 4-core machines, and you submit an application with -executor-memory 1G and --total-executor-cores 8. Then Spark will launch eight executors, each with 1 GB of RAM, on different machines. Spark does this by default to give applications a chance to achieve data locality for distributed filesystems running on the same machines (e.g., HDFS) because these systems typically have data spread out across all nodes.

I hope it helps!

whatsinthename
  • 1,828
  • 20
  • 59
  • how can we have multiple executors per node ..? Any setting or configuration for the same ..? – Maria Feb 13 '20 at 05:27
  • Its a cluster configuration. As per your cluster configuration, you can decide the values that need to be passed in spark-submit jobs. See the accepted answer in this link. It has a very good detailed explanation. https://stackoverflow.com/questions/37871194/how-to-tune-spark-executor-number-cores-and-executor-memory#2%20Answers and do accept the answer once you are satisfied – whatsinthename Feb 13 '20 at 05:33