How to create new rows in dataset based on multiple values present in array in one column of the dataset

Question

How to create new rows in dataset based on multiple values present in array in one column of the dataset:

I have a dataset with following data:

+----+---------+-------------------+------------------+
|name|productId|              total|            scores|
+----+---------+-------------------+------------------+
| aaa|      200|               0.29|            [0.29]|
| bbb|      200| 1.3900000000000001|      [0.53, 0.33]|
| aaa|      100|0.22999999999999998|      [0.12, 0.11]|
+----+---------+-------------------+------------------+

I want to transform this into below format in scala:

+----+---------+-------------------+------------------+
|name|productId|              total|            scores|
+----+---------+-------------------+------------------+
| aaa|      200|               0.29|            0.29  |
| bbb|      200| 1.3900000000000001|            0.53  |
| bbb|      200| 1.3900000000000001|            0.33  |
| aaa|      100|0.22999999999999998|            0.12  |
| aaa|      100|0.22999999999999998|            0.11  |
+----+---------+-------------------+------------------+

Very much similar to above, just the initial context was different. — Sprasad, Jul 22 '19 at 09:17

score 0 · Accepted Answer · answered Jul 22 '19 at 08:55

0

That's exactly what the explode function is meant for:

df.withColumn("score", explode('scores))

answered Jul 22 '19 at 08:55

Oli

9,766
5
25
46

Thanks Oli, that's what I was looking for. – Sprasad Jul 22 '19 at 09:16

How to create new rows in dataset based on multiple values present in array in one column of the dataset

1 Answers1