Scala - How to convert a pair RDD to an RDD?

Question

I have an RDD[Sale] and wanted to leave only the latest sales. So what I did is created a pair RDD and then performed grouping and filtering:

val sales: RDD[(String, Sale)] = rawSales.map(sale => sale.id -> sale)
      .groupByKey()
      .mapValues(_.maxBy(_.timestamp))

But how do I return back to RDD[Sale] instead of the pair RDD in this case?

The only way I figured out is the following:

val value: RDD[Sale] = sales.map(salePaired => salePaired._2)

Is it the most proper solution?

@pasha701 but I need and RDD after filtering transformations — samba, Oct 08 '18 at 13:29
As a performance hint, in this case, prefer reduceByKey instead of groupByKey/mapValues . https://stackoverflow.com/a/47516081/2131074 — GPI, Oct 08 '18 at 13:59
@GPI thanks. But I'm having trouble implementing `reduceByKey` with `maxBy` like this.. `.reduceByKey( ??? => _.maxBy(_.timestamp) )` — samba, Oct 08 '18 at 14:28
@GPI Could you give a hint on how to implement `reduceByKey` with `maxBy` correctly? — samba, Oct 08 '18 at 14:34
@samba something along the lines of `.reduceByKey((sale1, sale2) => if (sale1.timestamp > sale2.timestamp) sale1 else sale2)` — GPI, Oct 08 '18 at 15:04

score 1 · Answer 1 · answered Oct 08 '18 at 14:44

1

You can access the keys or values from pair RDD directly, like you access any Map

val keys: RDD[String] = sales.keys
val values: RDD[Sale] = sales.values

answered Oct 08 '18 at 14:44

1 Answers1