5

when I search off-heap in Spark configuration, there are two properties related(spark.executor.memoryOverhead and spark.memory.offHeap.size), I am not sure the relationship between these two.

If I enable spark.memory.offHeap.enabled, will spark.memory.offHeap.size be part of spark.executor.memoryOverhead? or these two types of off-heap memory are independent(thus the total off-heap memory is the sum of the two)

user7572131
  • 53
  • 1
  • 5
  • Does this answer your question? [spark off heap memory config and tungsten](https://stackoverflow.com/questions/43330902/spark-off-heap-memory-config-and-tungsten) – mazaneicha Apr 17 '20 at 18:12

1 Answers1

5

See my full answer here: https://stackoverflow.com/a/61723456/6470969

Short answer: as of current Spark version (2.4.5), if you specify spark.memory.offHeap.size, you should also add this portion to spark.executor.memoryOverhead. E.g. you set spark.memory.offHeap.size to 500M and you have spark.executor.memory=2G, then the default spark.executor.memoryOverhead is max(2*0.1, 384)=384M, but you'd better to increase the memoryOverhead to 384M+500M=884M.

Joshua Chen
  • 351
  • 3
  • 6