How to create a bag of words from a pandas dataframe

Question

Here's my dataframe

    CATEGORY    BRAND
0   Noodle  Anak Mas
1   Noodle  Anak Mas
2   Noodle  Indomie
3   Noodle  Indomie
4   Noodle  Indomie
23  Noodle  Indomie
24  Noodle  Mi Telor Cap 3
25  Noodle  Mi Telor Cap 3
26  Noodle  Pop Mie
27  Noodle  Pop Mie
...

I already make sure that df type is string, my code is

df = data[['CATEGORY', 'BRAND']].astype(str)
import collections, re
texts = df
bagsofwords = [ collections.Counter(re.findall(r'\w+', txt))
            for txt in texts]
sumbags = sum(bagsofwords, collections.Counter())

When I call

sumbags

The output is

 Counter({'BRAND': 1, 'CATEGORY': 1})

I want all of the data count in sumbags, except the title, to make it clear something like

Counter({'Noodle': 10, 'Indomie': 4, 'Anak': 2, ....}) # because it is bag of words

I need every 1 word counts

`Counter({'Noodle': 9, 'Indomie': 4, .....})` – Nabih Ibrahim Bawazir Sep 22 '17 at 08:57 — Nabih Ibrahim Bawazir, Sep 22 '17 at 08:57
I need 1 word counts – Nabih Ibrahim Bawazir Sep 22 '17 at 09:07 — Nabih Ibrahim Bawazir, Sep 22 '17 at 09:07
I do it manually, not by codes, thanks for your correction – Nabih Ibrahim Bawazir Sep 22 '17 at 09:10 — Nabih Ibrahim Bawazir, Sep 22 '17 at 09:10
This is what I want – Nabih Ibrahim Bawazir Sep 22 '17 at 09:11 — Nabih Ibrahim Bawazir, Sep 22 '17 at 09:11

score 9 · Accepted Answer · edited Mar 01 '18 at 14:19

IIUIC, use

Option 1] Numpy flatten and split

In [2535]: collections.Counter([y for x in df.values.flatten() for y in x.split()])
Out[2535]:
Counter({'3': 2,
         'Anak': 2,
         'Cap': 2,
         'Indomie': 4,
         'Mas': 2,
         'Mi': 2,
         'Mie': 2,
         'Noodle': 10,
         'Pop': 2,
         'Telor': 2})

Option 2] Use value_counts()

In [2536]: pd.Series([y for x in df.values.flatten() for y in x.split()]).value_counts()
Out[2536]:
Noodle     10
Indomie     4
Mie         2
Pop         2
Anak        2
Mi          2
Cap         2
Telor       2
Mas         2
3           2
dtype: int64

Options 3] Use stack and value_counts

In [2582]: df.apply(lambda x: x.str.split(expand=True).stack()).stack().value_counts()
Out[2582]:
Noodle     10
Indomie     4
Mie         2
Pop         2
Anak        2
Mi          2
Cap         2
Telor       2
Mas         2
3           2
dtype: int64

Details

In [2516]: df
Out[2516]:
   CATEGORY           BRAND
0    Noodle        Anak Mas
1    Noodle        Anak Mas
2    Noodle         Indomie
3    Noodle         Indomie
4    Noodle         Indomie
23   Noodle         Indomie
24   Noodle  Mi Telor Cap 3
25   Noodle  Mi Telor Cap 3
26   Noodle         Pop Mie
27   Noodle         Pop Mie

I prefer to use second solution, because easier to format pandas series to any pandas format — Nabih Ibrahim Bawazir, Sep 22 '17 at 09:19

How to create a bag of words from a pandas dataframe

1 Answers1

Linked