Pandas Melt with Multiple Value Vars

Question

I have a data set which is in wide format like this

   Index Country     Variable 2000 2001 2002 2003 2004 2005
   0     Argentina   var1     12   15   18    17  23   29
   1     Argentina   var2     1    3    2     5   7    5
   2     Brazil      var1     20   23   25   29   31   32
   3     Brazil      var2     0    1    2    2    3    3

I want to reshape my data to long so that year, var1, and var2 become new columns

  Index Country     year   var1 var2
  0     Argentina   2000   12   1
  1     Argentina   2001   15   3
  2     Argentina   2002   18   2
  ....
  6     Brazil      2000   20   0
  7     Brazil      2001   23   1

I got my code to work when I only had one variable by writing

df=(pd.melt(df,id_vars='Country',value_name='Var1', var_name='year'))

I cant figure out how to do this for a var1,var2, var3, etc.

score 20 · Accepted Answer · answered Jul 12 '17 at 20:31

Instead of melt, you can use a combination of stack and unstack:

(df.set_index(['Country', 'Variable'])
   .rename_axis(['Year'], axis=1)
   .stack()
   .unstack('Variable')
   .reset_index())

Variable    Country  Year  var1  var2
0         Argentina  2000    12     1
1         Argentina  2001    15     3
2         Argentina  2002    18     2
3         Argentina  2003    17     5
4         Argentina  2004    23     7
5         Argentina  2005    29     5
6            Brazil  2000    20     0
7            Brazil  2001    23     1
8            Brazil  2002    25     2
9            Brazil  2003    29     2
10           Brazil  2004    31     3
11           Brazil  2005    32     3

Scott Boston · Answer 2 · 2017-07-12T21:21:01.357

Option 1

Using melt then unstack for var1, var2, etc...

(df1.melt(id_vars=['Country','Variable'],var_name='Year')
    .set_index(['Country','Year','Variable'])
    .squeeze()
    .unstack()
    .reset_index())

Output:

Variable    Country  Year  var1  var2
0         Argentina  2000    12     1
1         Argentina  2001    15     3
2         Argentina  2002    18     2
3         Argentina  2003    17     5
4         Argentina  2004    23     7
5         Argentina  2005    29     5
6            Brazil  2000    20     0
7            Brazil  2001    23     1
8            Brazil  2002    25     2
9            Brazil  2003    29     2
10           Brazil  2004    31     3
11           Brazil  2005    32     3

Option 2

Using pivot then stack:

(df1.pivot(index='Country',columns='Variable')
   .stack(0)
   .rename_axis(['Country','Year'])
   .reset_index())

Output:

Variable    Country  Year  var1  var2
0         Argentina  2000    12     1
1         Argentina  2001    15     3
2         Argentina  2002    18     2
3         Argentina  2003    17     5
4         Argentina  2004    23     7
5         Argentina  2005    29     5
6            Brazil  2000    20     0
7            Brazil  2001    23     1
8            Brazil  2002    25     2
9            Brazil  2003    29     2
10           Brazil  2004    31     3
11           Brazil  2005    32     3

Option 3 (ayhan's solution)

Using set_index, stack, and unstack:

(df.set_index(['Country', 'Variable'])
   .rename_axis(['Year'], axis=1)
   .stack()
   .unstack('Variable')
   .reset_index())

Output:

Variable    Country  Year  var1  var2
0         Argentina  2000    12     1
1         Argentina  2001    15     3
2         Argentina  2002    18     2
3         Argentina  2003    17     5
4         Argentina  2004    23     7
5         Argentina  2005    29     5
6            Brazil  2000    20     0
7            Brazil  2001    23     1
8            Brazil  2002    25     2
9            Brazil  2003    29     2
10           Brazil  2004    31     3
11           Brazil  2005    32     3

I like how you've laid out the options in a very easy to read way :-) — piRSquared, Jul 12 '17 at 21:16

score 0 · Answer 3 · answered Jul 12 '17 at 21:59

numpy

years = df.drop(['Country', 'Variable'], 1)
y = years.values
m = y.shape[1]
c = df.Country.values
v = df.Variable.values

f0, u0 = pd.factorize(df.Country.values)
f1, u1 = pd.factorize(df.Variable.values)

w = np.empty((u1.size, u0.size, m), dtype=y.dtype)
w[f1, f0] = y

results = pd.DataFrame(dict(
        Country=u0.repeat(m),
        Year=np.tile(years.columns.values, u0.size),
    )).join(pd.DataFrame(w.reshape(-1, m * u1.size).T, columns=u1))

results

      Country  Year  var1  var2
0   Argentina  2000    12     1
1   Argentina  2001    15     3
2   Argentina  2002    18     2
3   Argentina  2003    17     5
4   Argentina  2004    23     7
5   Argentina  2005    29     5
6      Brazil  2000    20     0
7      Brazil  2001    23     1
8      Brazil  2002    25     2
9      Brazil  2003    29     2
10     Brazil  2004    31     3
11     Brazil  2005    32     3

Pandas Melt with Multiple Value Vars

3 Answers3

Option 1

Option 2

Option 3 (ayhan's solution)

Linked

Related