How to assuredly suppress a DeprecationWarning in Python?

Question

I believe this question was raised lots of times already but I have a specific use case where I can't solve the issue with many of the methods described on the web.

In one of my projects, I am using joblib library, and it shows DeprecationWarning because it uses imp library somewhere internally:

from sklearn.externals.joblib import Parallel, delayed

def main():
    xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
    print(sum(xs))

if __name__ == '__main__':
    main()

I am trying to filter out warning with interpreter option -W but it doesn't help:

$ python -W ignore example.py                                                                                                                   
[...]/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47:
DeprecationWarning: the imp module is deprecated in favour of importlib; 
see the module's documentation for alternative uses import imp
55

Also, I was trying an explicit filtering using warnings module but it is not helping also:

import warnings
warnings.simplefilter('ignore', category=DeprecationWarning)
from sklearn.externals.joblib import Parallel, delayed

def main():
    xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
    print(sum(xs))

if __name__ == '__main__':
    main()

I had similar issues with matplotlib module and some other third-party libraries. Probably there are some other ways (i.e., env vars) but I don't understand why these solutions don't work.

Could anyone explain how the warnings system actually works in Python? It is possible that third-party libraries intentionally override client's warnings settings? I would say that this question is one of the most obscure topics for me.

Answer 1

As requested, here's the answer as a separate post:

The trick is to use "with" warnings while importing sklearn (or the dependency that uses sklearn, in my case it was the hdbscan package):

with warnings.catch_warnings():
    # filter sklearn\externals\joblib\parallel.py:268:
    # DeprecationWarning: check_pickle is deprecated
    warnings.simplefilter("ignore", category=DeprecationWarning)
    import hdbscan

This will disable DeprecationWarning only for this module (because warnings-modification is attached to the with-block).

It is important to put this statement at the first position in your code where the module is imported, otherwise it won't work. E.g. if I was loading hdbscan in __init__.py, and the above code block appears in some sub-class that also loads hdbscan, I would still get the DeprecationWarning because Python ignores any subsequent import statement if the module/package is already loaded.

Therefore, it is important to check which modules/packages uses joblib\parallel.py and where those, from a linear code-perspective, are earliest loaded to the python object heap.

[EDIT]

As @devforfu points out in comments, the above solution does not work (anymore). I've looked again into this and since Python 3.7 DeprecationWarning is once again shown by default when triggered directly by code in __main__.. Furthermore, ignore warnings doesn't seem to work when a Dependency explicitly loads a depreciated module of some other package.

This is what appears to happen in my hdbscan example, which loads the depreciated modules sklearn.external.six and sklearn.externals.joblib.

Here's how to finally solve this annoying issue:

• make sure you've explicitly installed the standalone packages that are depreciated, e.g. conda install -c conda-forge joblib six
• create a fake import that will override the dependency import, e.g.:

try:
    sys.modules['sklearn.externals.six'] = __import__('six')
    sys.modules['sklearn.externals.joblib'] = __import__('joblib')
    import hdbscan
except ImportError:
    import hdbscan

If there's no import error, standalone six and joblib will be used. Otherwise, e.g. if a user hasn't installed six or joblib, the program will still work (because it loads both modules from sklearn.externals), but it will show the depreciation warning.

Answer 2

Interesting enough, that even following @Alex's advice, I still have warnings output, like this:

import warnings
with warnings.catch_warnings():
    warnings.simplefilter('ignore', category=DeprecationWarning)
    from sklearn.externals.joblib import Parallel, delayed

def main():
    xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
    print(sum(xs))

if __name__ == '__main__':
    main()

# $ python -W ignore example.py
# [...]
# DeprecationWarning: the imp module is deprecated in favour of importlib; 
# see the module's documentation for alternative uses
#  import imp
# 55

So eventually, I decided to do it in a very hacky way and disable all warnings because I am bit tired of looking for a proper way to deal with them. (Not only for this library, but for many others that seem to be very eager about bombarding you with non-suppressible warnings).

import warnings
def noop(*args, **kargs): pass
warnings.warn = noop
from sklearn.externals.joblib import Parallel, delayed

def main():
    xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
    print(sum(xs))

if __name__ == '__main__':
    main()

If I use @Alex's advice wrongly or some of you have a better solution, I would be glad to accept it as an answer.

---

Update 1

Ok, it seems that it is pretty hard to influence the warnings, raised somewhere internally in the package. So probably the most simple thing would be to just replace warnings.warn with noop, or maybe somehow import the internal dependencies in advance and suppress them with a context manager.

---

Update 2

Some time ago I found one more possible way to deal with warnings. You can redirect them into logging. In case if no logger is explicitly configured, these warnings are essentially suppressed. It works for Jupyter and some libraries I've tested.

import logging
logging.captureWarnings(True)