1

I'm trying to create a data frame with two date columns - one selected from another data frame and the other is to find the next Sunday after that date. But the issue I'm having is that I'm trying to use 7 - date_format(creation_date, 'u') to give me the number of days until the next Sunday and add it to the creation date like this:


df = df1.select(to_date(create_time), 
         to_date(date_add(df1.create_time, 7 - date_format(df1.create_time, 'u')))

However, this gives me a Column is not iterable error from date_add. Is there a solution around it?

ernest_k
  • 44,416
  • 5
  • 53
  • 99
Liumx31
  • 1,190
  • 1
  • 16
  • 33
  • 1
    Use `expr`: https://stackoverflow.com/questions/51140470/using-a-column-value-as-a-parameter-to-a-spark-dataframe-function – pault Sep 03 '20 at 01:41
  • `f.expr("date_add(to_date(create_time),8-dayofweek(date_time))")` – ernest_k Sep 03 '20 at 01:50

0 Answers0