Parametrize window by values in row in Pyspark

Asked Mar 30 '20 at 12:58

Active Dec 16 '20 at 12:13

Viewed 67 times

I would like to add a new column to my Pyspark dataframe using a Window function, where the rowsBetween are parametrized by values from columns from the dataframe.

I tried date_window.rowsBetween(-(F.lit(2) + offset), -offset), but Spark tells me ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions. which I did not expect in this case.

Is there any way to parametrize rowsBetween using values from specific columns?

asked Mar 30 '20 at 12:58

3yakuya

2,622
4
25
40

Can you give some sample input, so we can replicate the error? – DataBach Mar 30 '20 at 13:39
1

you cant put a col in rowsBetween or rangeBetween clauses. there might be a way a workaround if you could post your complete requirement – murtihash Mar 30 '20 at 16:15

Parametrize window by values in row in Pyspark

0 Answers0

Linked