4

Whenever I do a dse spark-submit <jarname>,it copies the jar in SPARK_WORKER_DIR (in my case /var/lib/spark-worker/worker-0). I want to get the jar automatically deleted once the spark job is successfully completed/run. Using this, I changed my SPARK_WORKER_OPTS in spark-env.sh as follows :

 export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800"

But the jar is still not getting deleted. Am I doing something wrong? What should I do?

Community
  • 1
  • 1
Amit Vade
  • 74
  • 6

1 Answers1

2

Adding this line to spark-env.sh and restarting the dse service worked for me:

export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=3600 -Dspark.worker.cleanup.appDataTtl=172800 "

I restarted the dse service by

nodetool drain
sudo service dse restart

This deletes the log 2 days after the job is complete.

user628904
  • 726
  • 1
  • 7
  • 20