Does collect_list() maintain relative ordering of rows?

Question

Imagine that I have the following DataFrame df:

+---+-----------+------------+
| id|featureName|featureValue|
+---+-----------+------------+
|id1|          a|           3|
|id1|          b|           4|
|id2|          a|           2|
|id2|          c|           5|
|id3|          d|           9|
+---+-----------+------------+

Imagine that I run:

df.groupBy("id")
  .agg(collect_list($"featureIndex").as("idx"),
       collect_list($"featureValue").as("val"))

Am I GUARANTEED that "idx" and "val" will be aggregated and keep their relative order? i.e.

GOOD                   GOOD                   BAD
+---+------+------+    +---+------+------+    +---+------+------+
| id|   idx|   val|    | id|   idx|   val|    | id|   idx|   val|
+---+------+------+    +---+------+------+    +---+------+------+
|id3|   [d]|   [9]|    |id3|   [d]|   [9]|    |id3|   [d]|   [9]|
|id1|[a, b]|[3, 4]|    |id1|[b, a]|[4, 3]|    |id1|[a, b]|[4, 3]|
|id2|[a, c]|[2, 5]|    |id2|[c, a]|[5, 2]|    |id2|[a, c]|[5, 2]|
+---+------+------+    +---+------+------+    +---+------+------+

NOTE: e.g. It's BAD because for id1 [a, b] should have been associated with [3, 4] (and not [4, 3]). Same for id2

score 14 · Accepted Answer · answered Jun 09 '17 at 04:12

I think you can rely on "their relative order" as Spark goes over rows one by one in order (and usually does not re-order rows if not explicitly needed).

If you are concerned with the order, merge these two columns using struct function before doing groupBy.

struct(colName: String, colNames: String*): Column Creates a new struct column that composes multiple input columns.

You could also use monotonically_increasing_id function to number records and use it to pair with the other columns (perhaps using struct):

monotonically_increasing_id(): Column A column expression that generates monotonically increasing 64-bit integers.

The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive.

So Spark DataFrame distribute the processing of the data 'horizontally' rather than 'vertically'. i.e. One executor will always receive the entire row and not only a few columns of it? — Marsellus Wallace, Jun 09 '17 at 20:05
Yes. Row is always in its entirety on one single executor/node. — Jacek Laskowski, Jun 09 '17 at 20:11

Does collect_list() maintain relative ordering of rows?

1 Answers1

Linked