Drop nested columns from dataframe in Pyspark

Asked Jun 21 '23 at 05:43

Active Jun 21 '23 at 12:03

Viewed 41 times

The nested schema can be struct of struct or array of struct or more complex and we are given a column name to drop from the dataframe.

Found this but the code is in scala and I don't have any knowledge of scala, this is exactly what I want

|-- details: array 
 |    |-- element: struct
 |    |    |-- source_id: string 
 |    |    |-- destination_id: string 
 |    |    |-- items: array 
 |    |    |    |-- element: struct 
 |    |    |    |    |-- data: array 
 |    |    |    |    |    |-- element: struct 
 |    |    |    |    |    |    |-- amount: integer
 |    |    |    |    |    |    |-- percent: double 
 |    |    |    |    |-- id: string 
 |    |    |    |    |-- period: struct 
 |    |    |    |    |    |-- end: long 
 |    |    |    |    |    |-- start: long

The schema is like this and the list contains [details.items.element.amount]. I want to automate the process of dropping columns from the list and the data inside dataframe should be kept.

edited Jun 21 '23 at 12:03

asked Jun 21 '23 at 05:43

user22106599

Does this answer your question? [Dropping nested column of Dataframe with PySpark](https://stackoverflow.com/questions/45061190/dropping-nested-column-of-dataframe-with-pyspark) – Prathik Kini Jun 21 '23 at 06:57
please add more details around your problem. it is always good to provide an [mcve] – samkart Jun 21 '23 at 07:13
sorry for not being clear, I provided the example – user22106599 Jun 21 '23 at 12:00
@Parthik Kini the codes in that link are working well with struct but if the column is of array type then they fail – user22106599 Jun 21 '23 at 12:10
struct of structs and array of structs will require different solutions -- very similar but not same. for array of structs, see the `transform` higher order function. – samkart Jun 21 '23 at 12:32
thank u...will try to implement this – user22106599 Jun 21 '23 at 15:50

Drop nested columns from dataframe in Pyspark

0 Answers0