Does Spark optimizes the DAG on encountering an action? I am aware that DAGScheduler pipelines transformations into a single stage. What I am trying to find is if there are any other optimization that are being applied on the DAG. And if so, in what situations are they applied(maybe you can point me to the rules or the code where it is explained).
Asked
Active
Viewed 486 times
1
-
https://stackoverflow.com/questions/25836316/how-dag-works-under-the-covers-in-rdd – thebluephantom Oct 03 '19 at 19:29
-
@thebluephantom I had already gone through that answer but it does not answer my question of optimization – A Beginner Oct 03 '19 at 19:44