I'm trying to apply a PandasUDFType.GROUPED_MAP function which takes a data frame as input and yields a data frame as an output. When I do sdf.groupby(key).apply(pandas_udf) does it apply the function parallelly to multiple groups based on available resources or sequentially one group after the other? I haven't changed any default settings of spark. What other alternatives can I employ if I want to execute udf on groups parallelly.
Asked
Active
Viewed 333 times
1 Answers
1
Yes UDFs are executed parallel but the execution is not as optimized as spark native functions.
More info here: Spark functions vs UDF performance?
uxke
- 446
- 3
- 9