Error using langdetect in python: "No features in text"

Question

Hey I have a csv with multilingual text. All I want is a column appended with a the language detected. So I coded as below,

from langdetect import detect 
import csv
with open('C:\\Users\\dell\\Downloads\\stdlang.csv') as csvinput:
with open('C:\\Users\\dell\\Downloads\\stdlang.csv') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
reader = csv.reader(csvinput)

    all = []
    row = next(reader)
    row.append('Lang')
    all.append(row)

    for row in reader:
        row.append(detect(row[0]))
        all.append(row)

    writer.writerows(all)

But I am getting the error as LangDetectException: No features in text

The traceback is as follows

runfile('C:/Users/dell/.spyder2-py3/temp.py', wdir='C:/Users/dell/.spyder2-py3')
Traceback (most recent call last):

  File "<ipython-input-25-5f98f4f8be50>", line 1, in <module>
    runfile('C:/Users/dell/.spyder2-py3/temp.py', wdir='C:/Users/dell/.spyder2-py3')

  File "C:\Users\dell\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Users\dell\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 89, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/dell/.spyder2-py3/temp.py", line 21, in <module>
    row.append(detect(row[0]))

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector_factory.py", line 130, in detect
    return detector.detect()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 136, in detect
    probabilities = self.get_probabilities()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 143, in get_probabilities
    self._detect_block()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 150, in _detect_block
    raise LangDetectException(ErrorCode.CantDetectError, 'No features in text.')

LangDetectException: No features in text.

This is how my csv looks like 1)skunkiest smokiest yummiest strain pain killer and mood lifter 2)Relaxation, euphorique, surélevée, somnolence, concentré, picotement, une augmentation de l’appétit, soulager la douleur Giggly, physique, esprit sédation 3)Reduzierte Angst, Ruhe, gehobener Stimmung, zerebrale Energie, Körper Sedierung 4)Calmante, relajante muscular, Relajación Mental, disminución de náuseas 5)重いフルーティーな幸せ非常に強力な頭石のバースト

Please help me with this.

you cannot read and write the same file! use another file for output! — Jean-François Fabre, Nov 24 '16 at 10:10
Could you include the first few lines of the CSV so we could take a look? — Haroldo_OK, Nov 24 '16 at 10:11
Perhaps you could also add line counting to the loop, so that, when the error happens, it will be possible to know exactly which of the rows caused it. — Haroldo_OK, Nov 24 '16 at 10:13
@Jean-FrançoisFabre tried with different file as output, but that doesn't work! — user7140275, Nov 24 '16 at 10:17
@Haroldo_OK please see to the edited question to have an idea of how my csv looks like. — user7140275, Nov 24 '16 at 10:32
On the Python console, you could try to manually call the `detect()` function for each of those strings to see which is causing the problem. — Haroldo_OK, Nov 24 '16 at 10:50

score 17 · Answer 1 · answered Oct 14 '17 at 04:30

17

You can use something like this to detect which line in your file is throwing the error:

for row in reader:
    try:
        language = detect(row[0])
    except:
        language = "error"
        print("This row throws and error:", row[0])
    row.append(language)
    all.append(row)

What you're going to see is that it probably fails at "重いフルーティーな幸せ非常に強力な頭石のバースト". My guess is that detect() isn't able to 'identify' any characters to analyze in that row, which is what the error implies.

Other things, like when the input is only a URL, also cause this error.

answered Oct 14 '17 at 04:30

Mark Cramer

2,614
5
33
57

Isn't the point of this package to detect languages that do not contain letters like Japanese, Chinese, etc? I don't get an error when running `detect('重いフルーティーな幸せ非常に強力な頭石のバースト')` – user3242036 Mar 17 '21 at 18:50
this is an old post, so maybe the package has been updated? when i ran it in 2017 that's where it failed. – Mark Cramer Mar 18 '21 at 00:07

score 7 · Answer 2 · answered Jul 14 '20 at 18:56

7

The error occurred when passing an object with no letters to detect. At least one letter should be there.

To reproduce, run any of below commands:

detect('.')
detect(' ')
detect('5')
detect('/')

So, you may apply some text pre-processing first to drop records in which row[0] value is an empty string,a null value, a white space, a number, a special character, or simply doesn't include any alphabets.

answered Jul 14 '20 at 18:56

Rola

1,598
13
12

This was my case, and one can simply check the string using `bool(re.match('^(?=.*[a-zA-Z])', your_string))` with `import re` to see whether there is at least one alphabet in your string – Pedram Dec 19 '20 at 11:21

Max Kleiner · Answer 3 · 2018-09-07T12:44:43.767

5

the problem is a null text or something like ' ' with no value; check this in a condition and loop your reader in a list comprehension or

from langdetect import detect   
textlang = [detect(elem) for elem in textlist if len(elem) > 50]

textlang = [detect(elem) if len(elem) > 50 else elem == 'no' for elem in textlist]

or with a loop

  texl70 = df5['Titletext']
  langdet = []                                                    

  for i in range(len(df5)):                                         
    try:                                                          
       lang=detect(texl70[i])                                      
    except:                                                       
       lang='no'                                                  
       print("This row throws error:", texl70[i])                 
    langdet.append(lang)

edited Sep 07 '18 at 12:44

answered Sep 06 '18 at 19:15

Max Kleiner

1,442
1
13
14

>>> textlang = [detect(elem) if len(elem) > 50 else elem == 'no' for elem in textlist] Its better to keep your list in sync and add a false if no lang detected – Max Kleiner Sep 06 '18 at 19:43
This error is not just for empty strings, but for text containing only an email address, URL, a sequence of numbers, etc. Checking these is a bit tricky, but error handling shown above should be helpful. – Oleg Melnikov Jan 14 '20 at 21:20
Right, there's less sense to detect lang in an email or URL, Thanks – Max Kleiner Jan 15 '20 at 12:35

Stranger16 · Answer 4 · 2022-08-04T07:36:09.223

0

The error occurrs when string has no letters. If you want to ignore that row and continue the process.

for i in df.index:
  str = df.iloc[i][1]  
  try:
    lang = detect(str)
  except:
    continue

edited Aug 04 '22 at 07:36

answered Aug 04 '22 at 07:30

Stranger16

1
1

score 0 · Answer 5 · answered Jul 05 '23 at 22:56

It is a bad practice to catch all possible exceptions. Let me propose something more complete, more readable and safer:

rx_letters = re.compile("[a-z]+", re.I)

for row in reader:
    try:
        if rx_letters.search(row[0]) is not None:
            row.append(detect(row[0]))
    except LangDetectException as e:
        row.append("?")
        print(f"Lang detect failed for: '{row[0]}'")

rx_letters check can be skipped, but I find it more elegant to check for the most basic condition.

Error using langdetect in python: "No features in text"

5 Answers5

Linked