0

this may be a very simple question. I want to transpose all the rows of dataframe to columns. I want to convert this df as shown below output DF. What are the ways in spark to achieve this?

Note : I have single column in input DF

import sparkSession.sqlContext.implicits._
val df = Seq(("row1"), ("row2"), ("row3"), ("row4"), ("row5")).toDF("COLUMN_NAME")
df.show(false)

    Input DF:
    +-----------+
    |COLUMN_NAME|
    +-----------+
    |row1       |
    |row2       |
    |row3       |
    |row4       |
    |row5       |
    +-----------+

    Output DF
    +----+----+----+----+----+
    |row1|row2|row3|row4|row5|
    +----+----+----+----+----+
rajesh
  • 172
  • 10
  • 1
    Does this answer your question? [How to pivot Spark DataFrame?](https://stackoverflow.com/questions/30244910/how-to-pivot-spark-dataframe) – mazaneicha Jun 20 '20 at 12:02
  • Think this answer will help: https://stackoverflow.com/a/49393080/1125159. It's not clear if you want row1, row2. etc to be column names in the output DataFrame. I'm guessing not, so you should update your question to include the desired column names. – Powers Jun 20 '20 at 13:03
  • Without ```groupBy```, ```pivot```, ```agg```, & ```first``` ..check this - https://stackoverflow.com/questions/61686883/scala-spark-flatten-array-of-key-value-structs/61733898#61733898 – Srinivas Jun 20 '20 at 13:48

1 Answers1

0

Does this help you ?

df.withColumn("group",monotonicallyIncreasingId  ).groupBy("group").pivot("COLUMN_NAME").agg(first("COLUMN_NAME")).show
abc_spark
  • 383
  • 3
  • 19