AWS ATHENA Transpose Columns into Rows

Question

I have a csv file uploaded to an S3 bucket which I pick up with AWS Glue then query using Athena. The CSV table is in the format below:

Item	Country	Category	2017	2018	2019	2020
Item1	CA	Network	128	129	130	129
Item2	CA	Desktop	128	129	130	129
Item3	CA	Apps	128	129	130	129

I want to convert that format into:

Item	Country	Category	Year	Value
Item1	CA	Network	2017	128
Item1	CA	Network	2018	129
Item1	CA	Network	2019	130
Item1	CA	Network	2020	129
Item2	CA	Desktop	2017	128
Item2	CA	Desktop	2018	129
Item2	CA	Desktop	2019	130
Item2	CA	Desktop	2020	129
Item3	CA	Apps	2017	128
Item3	CA	Apps	2018	129
Item3	CA	Apps	2019	130
Item3	CA	Apps	2020	129

How do I accomplish that using SQL in Athena?

I tried this but it doesn't work for me: Simple way to transpose columns and rows in SQL?

Any help is appreciated. Thanks!

Gordon Linoff · Answer 1 · 2021-02-03T01:52:14.320

2

You can do this with one scan by creating an array and then unnesting the array:

select t.item, t.country, t.category, r.year, r.value
from t cross join
     unnest( array[ cast(row(2017, t."2017") as row(year int, value int)),
                    cast(row(2018, t."2018") as row(year int, value int)),
                    cast(row(2019, t."2019") as row(year int, value int)),
                    cast(row(2020, t."2020") as row(year int, value int))
                  ]
           ) u(r);

If your table is really a view or complex query, the performance gain can be significant.

edited Feb 03 '21 at 01:52

answered Feb 02 '21 at 12:51

Gordon Linoff

1,242,037
58
646
786

Thanks for looking into this. I tried this but got the following error: SYNTAX_ERROR: line 3:21: Unknown type: row(year,value) – dribble Feb 03 '21 at 01:50
1

@dribble . . . I think it needs the type. – Gordon Linoff Feb 03 '21 at 01:52

score 1 · Accepted Answer · answered Feb 02 '21 at 08:54

Union all provides one option here:

SELECT Item, Country, Category, 2017 AS Year, "2017" AS Value FROM yourTable
UNION ALL
SELECT Item, Country, Category, 2018, "2018" FROM yourTable
UNION ALL
SELECT Item, Country, Category, 2019, "2019" FROM yourTable
UNION ALL
SELECT Item, Country, Category, 2020, "2020" FROM yourTable
ORDER BY Item, Country, Category, Year, Value;

This approach is not robust to having a dynamic number of year columns. But then again, you should not be going with that design anyway, since it is not normalized. So, hopefully you can use the above query, or a slight variant of it, to get your data normalized as it appears in the expected output.

Thanks! This works like a charm! The data is only updated once a year so this works perfectly! Thank you! — dribble, Feb 03 '21 at 02:02

AWS ATHENA Transpose Columns into Rows

2 Answers2