How to count number of columns in Spark Dataframe?

Question

I have this dataframe in Spark I want to count the number of available columns in it. I know how to count the number of rows in column but I want to count number of columns.

val df1 = Seq(
    ("spark", "scala",  "2015-10-14", 10,"rahul"),
    ("spark", "scala", "2015-10-15", 11,"abhishek"),
    ("spark", "scala", "2015-10-16", 12,"Jay"),
    ("spark","scala",null,13,"Kiran"))
  .toDF("bu_name","client_name","date","patient_id","paitent _name")
df1.show

Can anybody tell me how I can count number of column count in this dataframe? I am using the Scala language.

score 30 · Accepted Answer · answered Jul 27 '18 at 08:17

30

To count the number of columns, simply do:

df1.columns.size

answered Jul 27 '18 at 08:17

Shaido

27,497
23
70
73

Or df1.columns.length, seems to be faster – Johann Goulley Aug 28 '23 at 15:30
@JohannGoulley: They should be equivalent: https://stackoverflow.com/questions/22966705/scala-what-is-the-difference-between-size-and-length-of-a-seq – Shaido Aug 29 '23 at 01:10

score 10 · Answer 2 · edited Jan 25 '19 at 07:04

10

In python, the following code worked for me:

print(len(df.columns))

edited Jan 25 '19 at 07:04

Onema

7,331
12
66
102

answered Jan 25 '19 at 01:51

jillm_5

171
1
6

score 5 · Answer 3 · answered Oct 19 '19 at 08:45

data.columns accesses the list of column titles. All you have to do is count the number of items in the list. so

len(df1.columns)

works To obtain the whole data in a single variable, we do

rows = df.count()
columns = len(df.columns)
size = (rows, columns)
print(size)

score 1 · Answer 4 · answered Nov 16 '18 at 11:48

1

The length of the mutable indexed sequence also work.

df.columns.length

answered Nov 16 '18 at 11:48

Kris

1,618
1
13
13

score 0 · Answer 5 · answered May 09 '19 at 10:07

0

To count the columns of a Spark dataFrame:

len(df1.columns)

and to count the number of rows of a dataFrame:

df1.count()

answered May 09 '19 at 10:07

Saeid SOHEILY KHAH

747
3
10
23

score 0 · Answer 6 · answered Oct 23 '19 at 14:36

0

in Pyspark you can just result.select("your column").count()

answered Oct 23 '19 at 14:36

KeepLearning

517
7
10

How to count number of columns in Spark Dataframe?

6 Answers6