9

I'm using langdetect to determine the language of a set of strings which I know are either in English or French.

Sometimes, langdetect tells me the language is Romanian for a string I know is in French.

How can I make langdetect choose between English or French only, and not all other languages?

Thanks!

vandernath
  • 3,665
  • 3
  • 15
  • 24

2 Answers2

9

Option 1

One option would be using the package langid instead. Then you can simply restrict the languages with a method call:

import langid
langid.set_languages(['fr', 'en'])  # ISO 639-1 codes
lang, score = langid.classify('This is a french or english text')
print(lang) # en

Option 2

If you really want to use the langdetect package, you can copy the package folder (if you're not sure where it is, use python -m site --user-site) and remove the profiles you don't need from the folder langdetect\profiles.

This is not a very dynamic solution though.

Philip
  • 3,135
  • 2
  • 29
  • 43
5

The way I'd do this is to use detect_langs, which returns a list of Language objects with probabilities, and then iterate through this list, returning the language if one of the options is English or French, or None if this isn't the case. This function works well for this purpose:

from langdetect import detect_langs

def englishOrFrench(string):
    res = detect_langs(string)
    for item in res:
        if item.lang == "fr" or item.lang == "en":
            return item.lang
    return None

print(englishOrFrench("Bonjour"))              # fr
print(englishOrFrench("The quick brown fox"))  # en
print(englishOrFrench("Hallo, mein Freund"))   # None
Aaron Christiansen
  • 11,584
  • 5
  • 52
  • 78
  • 1
    lang detect kinda.. sucks. "38 HewcnonHe,!Me PlrIM HellaA.ne*aulee kicnon}le,wle npeAy(MOTpeHHbIX AorOBOpOM" is detected as en – thang Sep 18 '17 at 16:53
  • 2
    @thang And what output would you expect from a perfect tool? langdetect tries to fit every string to some language so if you use it to detect meaningless strings it breaks of course. – Jeyekomon Jan 05 '18 at 09:35
  • I think that was copied from a russian pdf doc. – thang Jan 05 '18 at 20:05
  • @thang Your example has almost nothing to do with Russian, it's a bogus copy output. – idle sign Oct 30 '21 at 09:47