4

I need to perform stemming on portuguese strings. To do so, i'm tokening the string using nltk.word_tokenize() function a then stemming each word individually. After that, I rebuild the string. It's working, but not performing well. How can i make it faster? The string length is about 2 million words.

    tokenAux=""
    tokens = nltk.word_tokenize(portugueseString)
        for token in tokens:
            tokenAux = token
            tokenAux = stemmer.stem(token)    
            textAux = textAux + " "+ tokenAux
    print(textAux)

Sorry for bad english and thanks!

yuridamata
  • 459
  • 1
  • 5
  • 13

3 Answers3

3

string is immutable so, it is not good practice to update string every time if the string is long. The link here explains various ways to concatenate string and shows performance analysis. And since, the iteration is done only once, it is good to choose generator expression over list comprehension. For details you can look into discussion here . Instead in this case, using generator expression with join can be helpful:

Using my_text for long string: len(my_text) -> 444399

Using timeit to compare:

%%timeit
tokenAux=""
textAux=""
tokens = nltk.word_tokenize(my_text)
for token in tokens:
    tokenAux = token
    tokenAux = stemmer.stem(token)    
    textAux = textAux + " "+ tokenAux

Result:

1 loop, best of 3: 6.23 s per loop

Using generator expression with join:

%%timeit 
' '.join(stemmer.stem(token) for token in nltk.word_tokenize(my_text))

Result:

1 loop, best of 3: 2.93 s per loop
niraj
  • 17,498
  • 4
  • 33
  • 48
1

String objects are immutable in Python. Look into your code:

textAux = ""
for token in tokens:
    # something important ...
    textAux = textAux + " "+ tokenAux

Every time you create a new string in a loop and assign it to textAux variable. This is not efficient.

I would store tokenAux elements in a list and just join them in the very end. See the example:

tokenAux = []  # we declare list for storing tokens
tokens = nltk.word_tokenize(portugueseString)
for token in tokens:
    tokenAux = token
    tokenAux = stemmer.stem(token)    
    textAux.append(tokenAux)  # we add new token into the resulting list

result = " ".join(textAux)  # join list using space as separator
print(result)

Compare the performance and share it with us :)

Useful links:

Viach Kakovskyi
  • 1,487
  • 1
  • 14
  • 21
0

You could read the string in as a text file and then perform the necessary operations to stem each word using PySpark. This will allow you to perform your operations in parallel.

You can also use the multiprocessing module.