0

I have a dataframe called clean, which is then separated in two samples: train_data and test_data, with the code below :

train_data = clean.sample(frac=0.75)
test_data = clean.drop(train_data.index)

I am trying to make a word frequency dataframe from the train_data dataframe. I started with the code

from collections import defaultdict as dct

phrases = []
for word in train_data['Message']:
    phrases.append(word.split())
    
ham = dct(int)
spam = dct(int)
    
for i in range(len(phrases)):
    if train_data['Category'][i] == 'ham':
        print(train_data['Category'][i])
    elif train_data['Category'][i] == 'spam':
        print(train_data['Category'][i])

But it gives me an error on the line with if train_data['Category'][i] == 'ham': when the index i is not in the train_data:

KeyError                                  Traceback (most recent call last)
~/Library/Python/3.8/lib/python/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3079             try:
-> 3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 5

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-97-17de52f682b3> in <module>
      9 
     10 for i in range(len(phrases)):
---> 11     if train_data['Category'][i] == 'ham':
     12         print(train_data['Category'][i])
     13     elif train_data['Category'][i] == 'spam':

~/Library/Python/3.8/lib/python/site-packages/pandas/core/series.py in __getitem__(self, key)
    851 
    852         elif key_is_scalar:
--> 853             return self._get_value(key)
    854 
    855         if is_hashable(key):

~/Library/Python/3.8/lib/python/site-packages/pandas/core/series.py in _get_value(self, label, takeable)
    959 
    960         # Similar to Index.get_value, but we do not fall back to positional
--> 961         loc = self.index.get_loc(label)
    962         return self.index._get_values_for_loc(self, loc, label)
    963 

~/Library/Python/3.8/lib/python/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3080                 return self._engine.get_loc(casted_key)
   3081             except KeyError as err:
-> 3082                 raise KeyError(key) from err
   3083 
   3084         if tolerance is not None:

KeyError: 5

The train_data looks like this (first 20 rows):

  Category                                            Message
1635      ham  you have come into my life and brought the sun...
3724      ham                    nothing splwat abt u and whr ru
1531      ham        oh dang i didnt mean o send that to you lol
1672     spam  urgent we are trying to contact u todays draw ...
2022     spam  u can win å100 of music gift vouchers every we...
4889      ham  sounds like there could be a lot of time spent...
4526      ham  understand his loss is my gain  so do you work...
1827      ham  hey gorgeous man my work mobile number is have...
3835      ham                   then ì_ come n pick me at 530 ar
342       ham                       where u been hiding stranger
2040      ham        you always make things bigger than they are
1788      ham                        arun can u transfr me d amt
860       ham                  in work now going have in few min
2298      ham  dont pick up d call when something important i...
763       ham  nothing but we jus tot u would ask cos u ba gu...
2475      ham                      mm i am on the way to railway
5156      ham  sir i need velusamy sirs date of birth and com...
164      spam  bangbabes ur order is on the way u should rece...
3671      ham   came to look at the flat seems ok in his 50s ...
4302      ham                                        yup im free

What is the issue?

James Z
  • 12,209
  • 10
  • 24
  • 44
Bogdan Rotaru
  • 3
  • 1
  • 4

2 Answers2

0

Look at the documentation for .loc and .iloc

You can try using if train_data['Category'].iloc[i] == 'ham'

Modified code will be:

for i in range(len(phrases)):
    if train_data['Category'].iloc[i] == 'ham':
        print(train_data['Category'].iloc[i])
    elif train_data['Category'].iloc[i] == 'spam':
        print(train_data['Category'].iloc[i])
ThePyGuy
  • 17,779
  • 5
  • 18
  • 45
0

KeyError: 5 indicates that the row with the index 5 does not exist. This happens because when using .sample(), the original DF's index is used and may have not picked the 5 row.

Example DF:

   letter
0     A
1     B
2     C
3     D
4     E
5     F

sampled = df.sample(frac=0.5)

   letter
3    D
1    B
4    E

If you try to iterate over sample using a for x in range(...), 0 is not present and will give an error.

You can use .reset_index() after .sample()

sampled = df.sample(frac=0.5).reset_index()

   letter
0    D
1    B
2    E

Anyways, Some tips:

  1. Don't iterate over rows of a DF. Try to use vectorized operations: How to iterate over rows in a DataFrame in Pandas

  2. To make a word frequency dict, you can use Counter from collections: https://docs.python.org/3/library/collections.html#collections.Counter

GusSL
  • 652
  • 7
  • 23