4

This gives me a frequency of words in a text:

 fullWords = re.findall(r'\w+', allText)

 d = defaultdict(int)

 for word in fullWords :
          d[word] += 1

 finalFreq = sorted(d.iteritems(), key = operator.itemgetter(1), reverse=True)

 self.response.out.write(finalFreq)

This also gives me useless words like "the" "an" "a"

My question is, is there a stop words library available in python which can remove all these common words? I want to run this on google app engine

hippietrail
  • 15,848
  • 18
  • 99
  • 158
demos
  • 2,630
  • 11
  • 35
  • 51

4 Answers4

5

You can download lists of stopwords as files in various formats, e.g. from here -- all Python needs to do is to read the file (and these are in csv format, easily read with the csv module), make a set, and use membership in that set (probably with some normalization, e.g., lowercasing) to exclude words from the count.

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
3

There's an easy way to handle this by slightly modifying the code you have (edited to reflect John's comment):

stopWords = set(['a', 'an', 'the', ...])
fullWords = re.findall(r'\w+', allText)
d = defaultdict(int)
for word in fullWords:
    if word not in stopWords:
        d[word] += 1
finalFreq = sorted(d.iteritems(), key=lambda t: t[1], reverse=True)
self.response.out.write(finalFreq)

This approach constructs the sorted list in two steps: first it filters out any words in your desired list of "stop words" (which has been converted to a set for efficiency), then it sorts the remaining entries.

David Z
  • 128,184
  • 27
  • 255
  • 279
  • 2
    Ummmm: why insert the stopwords and then rip them out again? Two lines to fix: ` if word not in stopwords: d[word] += 1` followed by a simple `finalFreq = d.items()` – John Machin Jul 04 '10 at 05:03
  • @John: I missed that. Although the number of stopwords is by definition limited, so it's not such a big deal. – David Z Jul 04 '10 at 05:21
  • re your latest edit: you don't need the `[]` (`sorted()` takes any iterable), and `(k,v) for k,v in d.iteritems()` is just `d.iteritems()` – John Machin Jul 04 '10 at 06:22
2

I know that NLTK has a package with a corpus and the stopwords for many languages, including English, see here for more information. NLTK has also a word frequency counter, it's a nice module for natural language processing that you should consider to use.

Tarantula
  • 19,031
  • 12
  • 54
  • 71
0
stopwords = set(['an', 'a', 'the']) # etc...
finalFreq = sorted((k,v) for k,v in d.iteritems() if k not in stopwords,
                      key = operator.itemgetter(1), reverse=True)

This will filter out any keys which are in the stopwords set.

Amber
  • 507,862
  • 82
  • 626
  • 550
  • See my comment on DavidZ's answer, yours has the same problem. – John Machin Jul 04 '10 at 05:05
  • It's not really a problem - performance wise, you're trading a set lookup for each resultant key for a set lookup for each word your regex matches. Which is more efficient will depend on the parameters of the problem set. You're already iterating over the set of result keys to output, anyways, so the generator expression for filtering doesn't involve much additional overhead - there's no extra lists being created, and the dict isn't being modified (so you're not actually "ripping them out"; just filtering them so that they never make it into the sorted list). – Amber Jul 04 '10 at 05:53