I want to sc.broadcast(val), the val maybe 100MB, my spark version is 2.3.2. I find the job will be cancelled by itself if I pass sc.broadcast(val).
I think it's because my val is large. So how to set maximum size of it?
Asked
Active
Viewed 193 times
0
rosefun
- 1,797
- 1
- 21
- 33
-
Please see this - https://stackoverflow.com/questions/41045917/what-is-the-maximum-size-for-a-broadcast-object-in-spark – dsk Jul 09 '20 at 09:20
-
and this - https://stackoverflow.com/questions/32021419/is-there-any-limit-on-size-of-a-spark-broadcast-variable – dsk Jul 09 '20 at 09:22
-
I see. Just use `spark.conf.set("spark.broadcast.blockSize", '100M')` on pyspark. – rosefun Jul 09 '20 at 09:40