Recursive version of 'reload'

Question

When I'm developing Python code, I usually test it in an ad-hoc way in the interpreter. I'll import some_module, test it, find a bug, fix the bug and save, and then use the built-in reload function to reload(some_module) and test again.

However, suppose that in some_module I have import some_other_module, and while testing some_module I discover a bug in some_other_module and fix it. Now calling reload(some_module) won't recursively re-import some_other_module. I have to either manually reimport the dependency (by doing something like reload(some_module.some_other_module), or import some_other_module; reload(some_other_module), or, if I've changed a whole bunch of dependencies and lost track of what I need to reload, I need to restart the entire interpreter.

What'd be more convenient is if there were some recursive_reload function, and I could just do recursive_reload(some_module) and have Python not only reload some_module, but also recursively reload every module that some_module imports (and every module that each of those modules imports, and so on) so that I could be sure that I wasn't using an old version of any of the other modules upon which some_module depends.

I don't think there's anything built in to Python that behaves like the recursive_reload function I describe here, but is there an easy way to hack such a thing together?

Answer 1

I've run up against the same issue, and you inspired me to actually solve the problem.

from types import ModuleType

try:
    from importlib import reload  # Python 3.4+
except ImportError:
    # Needed for Python 3.0-3.3; harmless in Python 2.7 where imp.reload is just an
    # alias for the builtin reload.
    from imp import reload

def rreload(module):
    """Recursively reload modules."""
    reload(module)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType:
            rreload(attribute)

Or, if you are using IPython, just use dreload or pass --deep-reload on startup.

Answer 2

I've run against the same issue and I've built up on @Mattew and @osa answer.

from types import ModuleType
import os, sys
def rreload(module, paths=None, mdict=None):
    """Recursively reload modules."""
    if paths is None:
        paths = ['']
    if mdict is None:
        mdict = {}
    if module not in mdict:
        # modules reloaded from this module
        mdict[module] = [] 
    reload(module)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType:
            if attribute not in mdict[module]:
                if attribute.__name__ not in sys.builtin_module_names:
                    if os.path.dirname(attribute.__file__) in paths:
                        mdict[module].append(attribute)
                        rreload(attribute, paths, mdict)
    reload(module)
    #return mdict

There are three differences:

1. In the general case, reload(module) has to be called at the end of the function as well, as @osa pointed out.
2. With circular import dependencies the code posted earlier would loop forever so I've added a dictionary of lists to keep track of the set of modules loaded by other modules. While circular dependencies are not cool, Python allows them, so this reload function deals with them as well.
3. I've added a list of paths (default is ['']) from which the reloading is allowed. Some modules don't like been reloaded the normal way, (as shown here).

Answer 3

The code worked great for dependency modules imported just as import another_module, but it failed when the module imported functions with from another_module import some_func.

I expanded on @redsk's answer to try and be smart about these functions. I've also added a blacklist because unfortunately typing and importlib don't appear in sys.builtin_module_names (maybe there are more). Also I wanted to prevent reloading of some dependencies I knew about.

I also track the reloaded module names and return them.

Tested on Python 3.7.4 Windows:

def rreload(module, paths=None, mdict=None, base_module=None, blacklist=None, reloaded_modules=None):
    """Recursively reload modules."""
    if paths is None:
        paths = [""]
    if mdict is None:
        mdict = {}
    if module not in mdict:
        # modules reloaded from this module
        mdict[module] = []
    if base_module is None:
        base_module = module
    if blacklist is None:
        blacklist = ["importlib", "typing"]
    if reloaded_modules is None:
        reloaded_modules = []
    reload(module)
    reloaded_modules.append(module.__name__)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType and attribute.__name__ not in blacklist:
            if attribute not in mdict[module]:
                if attribute.__name__ not in sys.builtin_module_names:
                    if os.path.dirname(attribute.__file__) in paths:
                        mdict[module].append(attribute)
                        reloaded_modules = rreload(attribute, paths, mdict, base_module, blacklist, reloaded_modules)
        elif callable(attribute) and attribute.__module__ not in blacklist:
            if attribute.__module__ not in sys.builtin_module_names and f"_{attribute.__module__}" not in sys.builtin_module_names:
                if sys.modules[attribute.__module__] != base_module:
                    if sys.modules[attribute.__module__] not in mdict:
                        mdict[sys.modules[attribute.__module__]] = [attribute]
                        reloaded_modules = rreload(sys.modules[attribute.__module__], paths, mdict, base_module, blacklist, reloaded_modules)
    reload(module)
    return reloaded_modules

Some notes:

1. I don't know why some builtin_module_names are prefixed with an underscore (for example collections is listed as _collections, so I have to do the double string check.
2. callable() returns True for classes, I guess that's expected but that was one of the reasons I had to blacklist extra modules.

At least now I am able to deep reload a module at runtime and from my tests I was able to go multiple levels deep with from foo import bar and see the result at each call to rreload()

(Apologies for the long and ugly depth, but the black formatted version doesn't look so readable on SO)

Answer 4

Wouldn't it be simpler to actually write some test cases and run them every time you are done with modifying your module?

What you are doing is cool (you are in essence using TDD (test driven development) but you are doing it wrong.

Consider that with written unit tests(using the default python unittest module, or better yet nose) you get to have tests that are reusable, stable and help you detect inconsitencies in your code much much faster and better than with testing your module in the interactive environment.

Answer 5

For Python 3.6+ you can use:

from types import ModuleType
import sys
import importlib

def deep_reload(m: ModuleType):
    name = m.__name__  # get the name that is used in sys.modules
    name_ext = name + '.'  # support finding sub modules or packages

    def compare(loaded: str):
        return (loaded == name) or loaded.startswith(name_ext)

    all_mods = tuple(sys.modules)  # prevent changing iterable while iterating over it
    sub_mods = filter(compare, all_mods)
    for pkg in sorted(sub_mods, key=lambda item: item.count('.'), reverse=True):
        importlib.reload(sys.modules[pkg])  # reload packages, beginning with the most deeply nested

Answer 6

I found the idea to just clear all the modules and then reimport your module here, which suggested to just do this:

import sys
sys.modules.clear()

This would mess up modules loaded that you don't want to reload (if you only want to reload your own modules). My idea is to only clear the modules that include your own folders. Something like this:

import sys
import importlib

def reload_all():
    delete_folders = ["yourfolder", "yourotherfolder"]

    for module in list(sys.modules.keys()):
        if any(folder in module for folder in delete_folders):
            del sys.modules[module]

    # And then you can reimport the file that you are running.
    importlib.import_module("yourfolder.entrypoint")

Reimporting your entry point will reimport all of its imports since the modules were cleared and it's automatically recursive.

Answer 7

Technically, in each file you could put a reload command, to ensure that it reloads each time it imports

a.py:

def testa():
    print 'hi!'

b.py:

import a
reload(a)
def testb():
    a.testa()

Now, interactively:

import b
b.testb()
#hi!

#<modify a.py>

reload(b)
b.testb()
#hello again!

Answer 8

I found the answer of redsk very useful. I propose a simplified (for the user, not as code) version where the path to the module is automatically gathered and recursion works for an arbitrary number of levels. Everything is self-contained in a single function. Tested on Python 3.4. I guess for python 3.3 one must import reload from imp instead of ... from importlib. It also checks if the __file__ file is present, which might be false if the coder forgets to define an __init__.py file in a submodule. In such case, an exception is raised.

def rreload(module):
    """
    Recursive reload of the specified module and (recursively) the used ones.
    Mandatory! Every submodule must have an __init__.py file
    Usage:
        import mymodule
        rreload(mymodule)

    :param module: the module to load (the module itself, not a string)
    :return: nothing
    """

    import os.path
    import sys

    def rreload_deep_scan(module, rootpath, mdict=None):
        from types import ModuleType
        from importlib import reload

        if mdict is None:
            mdict = {}

        if module not in mdict:
            # modules reloaded from this module
            mdict[module] = []
        # print("RReloading " + str(module))
        reload(module)
        for attribute_name in dir(module):
            attribute = getattr(module, attribute_name)
            # print ("for attr "+attribute_name)
            if type(attribute) is ModuleType:
                # print ("typeok")
                if attribute not in mdict[module]:
                    # print ("not int mdict")
                    if attribute.__name__ not in sys.builtin_module_names:
                        # print ("not a builtin")
                        # If the submodule is a python file, it will have a __file__ attribute
                        if not hasattr(attribute, '__file__'):
                            raise BaseException("Could not find attribute __file__ for module '"+str(attribute)+"'. Maybe a missing __init__.py file?")

                        attribute_path = os.path.dirname(attribute.__file__)

                        if attribute_path.startswith(rootpath):
                            # print ("in path")
                            mdict[module].append(attribute)
                            rreload_deep_scan(attribute, rootpath, mdict)

    rreload_deep_scan(module, rootpath=os.path.dirname(module.__file__))

Answer 9

Below is the recursive reload function that I use, including a magic function for ipython/jupyter.

It does a depth-first search through all sub-modules and reloads them in the correct order of dependence.

import logging
from importlib import reload, import_module
from types import ModuleType
from IPython.core.magic import register_line_magic

logger = logging.getLogger(__name__)


def reload_recursive(module, reload_external_modules=False):
    """
    Recursively reload a module (in order of dependence).

    Parameters
    ----------
    module : ModuleType or str
        The module to reload.

    reload_external_modules : bool, optional

        Whether to reload all referenced modules, including external ones which
        aren't submodules of ``module``.

    """
    _reload(module, reload_external_modules, set())


@register_line_magic('reload')
def reload_magic(module):
    """
    Reload module on demand.

    Examples
    --------
    >>> %reload my_module
    reloading module: my_module

    """
    reload_recursive(module)


def _reload(module, reload_all, reloaded):
    if isinstance(module, ModuleType):
        module_name = module.__name__
    elif isinstance(module, str):
        module_name, module = module, import_module(module)
    else:
        raise TypeError(
            "'module' must be either a module or str; "
            f"got: {module.__class__.__name__}")

    for attr_name in dir(module):
        attr = getattr(module, attr_name)
        check = (
            # is it a module?
            isinstance(attr, ModuleType)

            # has it already been reloaded?
            and attr.__name__ not in reloaded

            # is it a proper submodule? (or just reload all)
            and (reload_all or attr.__name__.startswith(module_name))
        )
        if check:
            _reload(attr, reload_all, reloaded)

    logger.debug(f"reloading module: {module.__name__}")
    reload(module)
    reloaded.add(module_name)

Answer 10

Building on top of @machinekoder's answer above, I'd set the reloaded modules to sys and finally return the reloaded module m from sys like so:

def deep_reload(m: ModuleType):
    name = m.__name__  # get the name that is used in sys.modules
    name_ext = name.split('.')[0]  # support finding submodules or packages

    def compare(loaded: str):
        return (loaded == name) or loaded.startswith(name_ext)

    all_mods = tuple(sys.modules)  # prevent changing iterable while iterating over it
    sub_mods = filter(compare, all_mods)
    for pkg in sorted(sub_mods, key=lambda item: item.count('.'), reverse=True):
        sys.modules[pkg] = importlib.reload(sys.modules[pkg])
    return sys.modules[name]

Answer 11

It is a tricky thing to do - I have an working example in this answer: how to find list of modules which depend upon a specific module in python