1

I have a table like that:

enter image description here

In this table,artist_id stands for a particular singer, Ds is a date(from 2015 Mar 1st to the end of Apr) and like is how many people liked this singer's songs in this particular day. I want to get the accumulated value of like, for example, in the day 20150303, the value will be the sum of original value of 20150301 and 20150302 and 20150303. How can I make it?

Liu Chong
  • 313
  • 3
  • 4
  • 14

1 Answers1

1

You can use the aggregate functions provided by spark and get the output.

Your question says, based on time, but as per schema, its actually a column of date, hence you aggregate on Ds and get sum of like similar to

df.groupBy("Ds").sum("like")

Update: To get the sum of all days previous to provided date, Reduce can be used after applying filter for the provided date to fetch results of this and previous date and then summing up all using reduce or aggregate function sum

more details can be found here

Community
  • 1
  • 1
vinay
  • 1,366
  • 1
  • 13
  • 23
  • groupBy'Ds' and then sum can only get a single day's data sum, but I want to sum data of all days before a day and this day. – Liu Chong Apr 04 '17 at 08:30