how to count categorical values including zero occurrence?

Question

I want to count number of code by month. This is my example dataframe.

        id    month  code
0     sally    0  s_A
1     sally    0    s_B
2     sally    0   s_C
3     sally    0   s_D
4     sally    0    s_E
5     sally    0   s_A
6     sally    0    s_A
7     sally    0   s_B
8     sally    0   s_C
9     sally    0   s_A

I transformed to this Series using count().

df.groupby(['id', 'code', 'month']).month.count()

id      code   month  count
sally  s_A      0    12
                1    10
                2     3
                7    15

But, I want to include zero occurrence, like this.

id      code   month  count
sally  s_A      0    12
                1    10
                2     3
                3    0
                4    0
                5    0
                6    0
                7    15
                8    0
                9    0
                10   0
                11   0

you didn't show the `'m'` column – splinter Apr 11 '17 at 07:52 — splinter, Apr 11 '17 at 07:52
@splinter I updated. m is month. – planaria Apr 11 '17 at 07:54 — planaria, Apr 11 '17 at 07:54

score 1 · Accepted Answer · edited Sep 27 '17 at 16:47

You can use reindex with new Multindex created from_product:

Notice:
What is the difference between size and count in pandas?

df = pd.DataFrame({
'month': [0, 0, 0, 0, 1, 1, 1, 2, 2, 7], 
'code': ['s_A', 's_A', 's_A', 's_A', 's_A', 's_A', 's_A', 's_B', 's_B', 's_B'], 
'id': ['sally1','sally1','sally1','sally','sally','sally','sally','sally','sally','sally']})

print (df)
  code      id  month
0  s_A  sally1      0
1  s_A  sally1      0
2  s_A  sally1      0
3  s_A   sally      0
4  s_A   sally      1
5  s_A   sally      1
6  s_A   sally      1
7  s_B   sally      2
8  s_B   sally      2
9  s_B   sally      7

df = df.groupby(['id', 'code', 'month']).size()
n = ['id','code','month']
mux = pd.MultiIndex.from_product([df.index.levels[0],df.index.levels[1], range(13)], names=n)
df = df.reindex(mux, fill_value=0)
print (df)
id      code  month
sally   s_A   0        1
              1        3
              2        0
              3        0
              4        0
              5        0
              6        0
              7        0
              8        0
              9        0
              10       0
              11       0
              12       0
        s_B   0        0
              1        0
              2        2
              3        0
...
...

Old solution:

reindex with unstack and stack, but then need some data cleaning:

df = df.groupby(['id', 'code', 'month']).size() \
       .to_frame('count') \
       .unstack([0,1], fill_value=0) \
       .reindex(range(13), fill_value=0) \
       .stack([1,2], dropna=False) \
       .fillna(0) \
       .astype(int) \
       .swaplevel(0,2) \
       .sort_index()
print (df)
                   count
code id     month       
s_A  sally  0          1
            1          3
            2          0
            3          0
            4          0
            5          0
            6          0
            7          0
            8          0
            9          0
            10         0
            11         0
            12         0
     sally1 0          3
            1          0
            2          0
            3          0
            4          0

Mux is great solution. You saved my day! – planaria Apr 11 '17 at 08:33 — planaria, Apr 11 '17 at 08:33

how to count categorical values including zero occurrence?

1 Answers1

Linked