How to stack several 1 x 1 dataframes to get a 1 x n dataframe?

Question

I have a huge dataframe containing millions of rows. From these rows I derive new k dataframes which have only 1 row and 1 column. What's a good way to concatenate these k dataframes together so as to now get a a dataframe 1 x k that has 1 row and k columns.

1. In the past I started with using a crossJoin among all the k dataframes, such as df1.crossJoin(df2).crossJoin(df3).crossJoin(dfk)

   This resulted in a broadcast timeout error,

2. Later I moved to what I thought is a smarter solutions.

   df1.withColumn("temp_id", lit(0)).join(df2.withColumn("temp_id", lit(0)), "temp_id").drop("temp_id").

   This resulted in a weirder yet similar error of broadcast timeout.

The result that I really want is a new DataFrame with 1 row and k columns which in numpy/pandas language could be

pandas.concat(..., axis=1) OR np.vstack(.....)

Answer 1

The operation I think you want to perform is a "zip" operation. Spark does not provide this method for Dataframes, but you can see how it works with the following example (A Spark version follows this example):

scala> val l1 = List("a", "b")
l1: List[String] = List(a, b)

scala> val l2 = List(1,2)
l2: List[Int] = List(1, 2)

scala> val zipped = l1.zip(l2)
zipped: List[(String, Int)] = List((a,1), (b,2))

scala> zipped.foreach(println)
(a,1)
(b,2)

scala> 

How to do this in Spark has already been answered here: How to zip two (or more) DataFrame in Spark

Basically, you do this:

val zippedRDD = df1.rdd.zip(df2.rdd)

this will leave you with an RDD, which you can convert to a DF or DS as needed in the usual way.