-2

I am very much new to scala and help regarding date format. I have a dataframe :

+-----+----------+----------+-----+                                             
| name| startDate|   endDate|price|
+-----+----------+----------+-----+
|steak|01/01/1999|01/01/2000|  150|
|steak|02/02/2000|13/01/2000|  180|
| fish|03/03/2000|12/01/2000|  100|
+-----+----------+----------+-----+

I need to convert the enddate column to yyyyMMdd format and need the result like below:

+-----+----------+----------+-----+                                             
| name| startDate|   endDate|price|
+-----+----------+----------+-----+
|steak|01/01/1999|20000101  |  150|
|steak|02/02/2000|20000113  |  180|
| fish|03/03/2000|20000112  |  100|
+-----+----------+----------+-----+

I have tried the below code and getting "NULL" in EndDate column

val result  = df.withColumn("EndDate",date_format(col("endDate"), "yyyyMMdd")).select("*").show()

Can anyone please help me?

abssab
  • 103
  • 1
  • 3
  • 10

1 Answers1

1

For Spark 2.2+

You should convert to DateType for the column and change the format as below 

df.withColumn("EndDate",
  date_format(to_date(col("endDate"),"MM/dd/yyyy"), "yyyyMMdd")
)

EDIT For Spark < 2.2 use unix_timestamp

df.withColumn("date",
  date_format(unix_timestamp(col("endDate"),"MM/dd/yyyy").cast(TimestampType), "yyyyMMdd")
)

The one that doesnot match the format returns null, As 13/01/2000 is invalid for the format MM/dd/yyyy which gives you null

koiralo
  • 22,594
  • 6
  • 51
  • 72
  • i have tried the suggestion and it is saying too many arguments for method to_date: (e: org.apache.spark.sql.Column)org.apache.spark.sql.Column – abssab Jul 30 '19 at 07:51
  • Can you share how did you use it ? – koiralo Jul 30 '19 at 07:52
  • val result = df.select(date_format(to_date(col("enddate"),"MM/dd/yyyy"), "yyyyMMdd")).show – abssab Jul 30 '19 at 07:54
  • That looks totally fine to me, can you share the full error? or did you import `import org.apache.spark.sql.functions._`, What version of spark are you using ? – koiralo Jul 30 '19 at 07:56