Pyspark - Add column with number of occurences

Asked Sep 13 '22 at 19:27

Active Sep 13 '22 at 19:27

Viewed 110 times

Let's say I have a dataframe with a column A and I would like to aggregate another column with the occurence of the distinct values in A without resulting in a grouped dataframe, I mean I would like to have the following:

A	A_counts
a	3
a	3
b	1
c	2
c	2
a	3

Thanks in advance

asked Sep 13 '22 at 19:27

pmolok

What is your expected output? – Jonathan Lam Sep 14 '22 at 02:22
use the `count` window function with an arbitrary partition col. see [this example](https://stackoverflow.com/questions/60672901/get-total-row-count-over-a-window) – samkart Sep 14 '22 at 04:41
1

`groupBy` + `count` then join with the original dataframe with `A` as key – philantrovert Sep 14 '22 at 11:46
@philantrovert 's answer solved my problem! `df.join(df.groupBy("A").count(), ["A"], 'left').withColumnRenamed("count","A_counts")` Thank you very much!! – pmolok Sep 27 '22 at 21:10

Pyspark - Add column with number of occurences

0 Answers0