How to properly handle a circular module dependency in Python?

Question

Trying to find a good and proper pattern to handle a circular module dependency in Python. Usually, the solution is to remove it (through refactoring); however, in this particular case we would really like to have the functionality that requires the circular import.

EDIT: According to answers below, the usual angle of attack for this kind of issue would be a refactor. However, for the sake of this question, assume that is not an option (for whatever reason).

The problem:

The logging module requires the configuration module for some of its configuration data. However, for some of the configuration functions I would really like to use the custom logging functions that are defined in the logging module. Obviously, importing the logging module in configuration raises an error.

The possible solutions we can think of:

1. Don't do it. As I said before, this is not a good option, unless all other possibilities are ugly and bad.

2. Monkey-patch the module. This doesn't sound too bad: load the logging module dynamically into configuration after the initial import, and before any of its functions are actually used. This implies defining global, per-module variables, though.

3. Dependency injection. I've read and run into dependency injection alternatives (particularly in the Java Enterprise space) and they remove some of this headache; however, they may be too complicated to use and manage, which is something we'd like to avoid. I'm not aware of how the panorama is about this in Python, though.

What is a good way to enable this functionality?

Thanks very much!

Answer 1

As already said, there's probably some refactoring needed. According to the names, it might be ok if a logging modules uses configuration, when thinking about what things should be in configuration one think about configuration parameters, then a question arises, why is that configuration logging at all?

Chances are that the parts of the code under configuration that uses logging does not belong to the configuration module: seems like it is doing some kind of processing and logging either results or errors.

Without inner knowledge, and using only common sense, a "configuration" module should be something simple without much processing and it should be a leaf in the import tree.

Hope it helps!

Answer 2

Will this work for you?

# MODULE a (file a.py)
import b
HELLO = "Hello"

# MODULE b (file b.py)
try:
    import a
    # All the code for b goes here, for example:
    print("b done",a.HELLO))
except:
    if hasattr(a,'HELLO'):
        raise
    else:
        pass

Now I can do an import b. When the circular import (caused by the import b statement in a) throws an exception, it gets caught and discarded. Of course your entire module b will have to indented one extra block spacing, and you have to have inside knowledge of where the variable HELLO is declared in a.

If you don't want to modify b.py by inserting the try:except: logic, you can move the whole b source to a new file, call it c.py, and make a simple file b.py like this:

# new Module b.py
try:
    from c import *
    print("b done",a.HELLO) 
except:
    if hasattr(a,"HELLO"):
        raise
    else:
        pass

# The c.py file is now a copy of b.py:
import a
# All the code from the original b, for example:
print("b done",a.HELLO))

This will import the entire namespace from c to b, and paper over the circular import as well.

I realize this is gross, so don't tell anyone about it.

Answer 3

A cyclic module dependency is usually a code smell.

It indicates that part of the code should be re-factored so that it is external to both modules.

Answer 4

So if I'm reading your use case right, logging accesses configuration to get configuration data. However, configuration has some functions that, when called, require that stuff from logging be imported in configuration.

If that is the case (that is, configuration doesn't really need logging until you start calling functions), the answer is simple: in configuration, place all the imports from logging at the bottom of the file, after all the class, function and constant definitions.

Python reads things from top to bottom: when it comes across an import statement in configuration, it runs it, but at this point, configuration already exists as a module that can be imported, even if it's not fully initialized yet: it only has the attributes that were declared before the import statement was run.

I do agree with the others though, that circular imports are usually a code smell.