Avoiding circular imports when base class returns a subclass instance in an importable module

Question

Summary

TLDR: How to avoid circular import errors when a base class returns a subclass instance in an importable module?

I have collected some solutions from other locations/questions (see A-D, below) but none is satisfactory IMHO.

Starting point

Based on this and this question, I have the following hypothetical working example as a starting point:

# onefile.py

from abc import ABC, abstractmethod

class Animal(ABC):
    def __new__(cls, weight: float):
        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in [Dog, Cat]:
                try:
                    return subcls(weight)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)

    @property
    @abstractmethod
    def weight(self) -> float:
        """weight of the animal in kg."""
        ...


class Dog(Animal):
    def __init__(self, weight: float = 5):
        if not (1 < weight < 90):
            raise ValueError("No dog has this weight")
        self._weight = weight

    weight: float = property(lambda self: self._weight)


class Cat(Animal):
    def __init__(self, weight: float = 5):
        if not (0.5 < weight < 15):
            raise ValueError("No cat has this weight")
        self._weight = weight

    weight: float = property(lambda self: self._weight)


if __name__ == "__main__":

    a1 = Dog(34)
    try:
        a2 = Dog(0.9)  # ValueError
    except ValueError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

    a3 = Cat(0.8)
    try:
        a4 = Cat(25)  # ValueError
    except ValueError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

    a5 = Animal(80)  # can only be dog; should return dog.
    assert type(a5) is Dog
    a6 = Animal(0.7)  # can only be cat; should return cat.
    assert type(a6) is Cat
    a7 = Animal(10)  # can be both; should return dog.
    assert type(a7) is Dog
    try:
        a8 = Animal(400)
    except NotImplementedError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

This file runs correctly.

Refactor into importable module, in separate files

I want to have Cat, Dog and Animal as importable classes from the module zoo. To that end, I create a folder zoo, with the files animal.py, dog.py, cat.py, and __init__.py. The file usage.py is kept in parent folder. This is what these files look like:


# zoo/animal.py

from abc import ABC, abstractmethod
from .dog import Dog
from .cat import Cat

class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in [Dog, Cat]:
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)

    @property
    @abstractmethod
    def weight(self) -> float:
        """weight of the animal in kg."""
        ...


# zoo/dog.py

from .animal import Animal

class Dog(Animal):
    def __init__(self, weight: float = 5):
        if not (1 < weight < 90):
            raise ValueError("No dog has this weight")
        self._weight = weight

    weight: float = property(lambda self: self._weight)


# zoo/cat.py

from .animal import Animal

class Cat(Animal):
    def __init__(self, weight: float = 5):
        if not (0.5 < weight < 15):
            raise ValueError("No cat has this weight")
        self._weight = weight

    weight: float = property(lambda self: self._weight)


# zoo/__init__.py

from .dog import Dog
from .cat import Cat
from .animal import Animal


# usage.py
  
from zoo import Dog, Cat, Animal

a1 = Dog(34)
try:
    a2 = Dog(0.9)  # ValueError
except ValueError:
    pass
else:
    raise RuntimeError("Should have raised Exception!")

a3 = Cat(0.8)
try:
    a4 = Cat(25)  # ValueError
except ValueError:
    pass
else:
    raise RuntimeError("Should have raised Exception!")

a5 = Animal(80)  # can only be dog; should return dog.
assert type(a5) is Dog
a6 = Animal(0.7)  # can only be cat; should return cat.
assert type(a6) is Cat
a7 = Animal(10)  # can be both; should return dog.
assert type(a7) is Dog
try:
    a8 = Animal(400)
except NotImplementedError:
    pass
else:
    raise RuntimeError("Should have raised Exception!")

This is what is currently not working; the refactoring reintroduces the ImportError (...) (most likely due to a circular import). The problem is that animal.py references dog.py and cat.py, and vice versa.

Possible solutions

Some possibilities are available (some taken from the linked question); here are some options. The code samples only show how relevant parts of the files change.

A: Import modules and move to after `Animal` class definition

from abc import ABC, abstractmethod

class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in [dog.Dog, cat.Cat]:  # <-- instead of [Dog, Cat]
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)

    (...)

from . import dog  # <-- import module instead of class, and import at end, to avoid circular import error
from . import cat  # <-- same

This works.

Disadvantages:

From dog.py, the Dog class is really only needed. It's confusing that it's imported completely (though this is considered best practices by some).
Bigger issue: the imports need to be placed at the end of the file, which is definitely bad practice.

B: Move imports inside function

# zoo/animal.py

from abc import ABC, abstractmethod

class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        from .dog import Dog  # <-- imports here instead of at module level
        from .cat import Cat  # <-- imports here instead of at module level

        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in [Dog, Cat]:
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)
  
    (...)

This works too.

Disadvantages:

Goes against best practice of module-level-only imports.
If Dog or Cat are needed at several locations, the imports need to be repeated.

C: Remove imports and find class by name

# zoo/animal.py

from abc import ABC, abstractmethod

class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        if cls is Animal:
            # Try to return subclass instance instead.
            subclasses = {sc.__name__: sc for sc in Animal.__subclasses__()}  # <-- create dictionary
            for subcls in [subclasses["Dog"], subclasses["Cat"]]:   # <-- instead of [Dog, Cat]
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)
   
   (...)

This also works. In order to avoid creating the dictionary every single time, a registry as shown in this answer could be used as well.

Disadvantages:

Verbose and barely readable.
If class name changes, this code breaks.

D: Update dummy variable

# zoo/animal.py

from abc import ABC, abstractmethod

_Dog = _Cat = None  # <-- dummies, to be assigned by subclasses.


class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in [_Dog, _Cat]:  # <-- instead of [Dog, Cat]
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)
    
    (...)

# zoo/dog.py

from . import animal
from .animal import Animal


class Dog(Animal):
    (...)


animal._Dog = Dog  # <-- update protected variable

# zoo/cat.py analogously

This works as well.

Disadvantages:

Unclear to reader, what the _Dog and _Cat variables in zoo/animal.py represent.
Coupling between files; change/use of module's "protected" variables from outside.

E: a better solution??

None of A-D is satisfying, in my opinion, and I'm wondering if there's another way. This is where you come in. ;) There might not be another way - in that case I'm curious to hear what your preferred approach would be, and why.

Many thanks

First of all, your current `Animal` class returns an instance of one of its known subclasses. So only if the module containing `Dog` has been loaded, then the `Dog` class will be available. Is it enough for your requirements or do you want all the subclasses to be available from `Animal`? — Serge Ballesta, Feb 01 '22 at 12:28
Thanks for your comment @SergeBallesta, but I'm not sure I'm following. I must somehow make `Dog` available to from within `animal.py`, otherwise it will not know what the name '`Dog`' means. (To answer your question - in this case only a subset of all possible subclasses must be available to `Animal`, and they must be checked in a specific order.) — ElRudi, Feb 01 '22 at 12:52
I'm using solution B thanks to you, though it's not very satisfying... — Kiruahxh, May 27 '22 at 14:06

score 2 · Answer 1 · answered Feb 01 '22 at 13:30

IMHO you only need a simple package, and do the appropriate initialization in the __init__.py file:

Overall structure:

zoo folder accessible from the Python path
| __init__.py
| animal.py
| dog.py
| cat.py
|  other files...

animal.py - no direct dependency on any other module

from abc import ABC, abstractmethod

subclasses = []    # will be initialized from the package __init__ file

class Animal(ABC):
    def __new__(cls, *args, **kwargs):
        if cls is Animal:
            # Try to return subclass instance instead.
            for subcls in subclasses:
                try:
                    return subcls(*args, **kwargs)
                except ValueError:
                    pass
            raise NotImplementedError("No appropriate subclass found.")
        return super().__new__(cls)

    @property
    @abstractmethod
    def weight(self) -> float:
        """weight of the animal in kg."""
        ...

dog.py - depends on animal

from .animal import Animal

class Dog(Animal):
    def __init__(self, weight: float = 5):
        if not (1 < weight < 90):
            raise ValueError("No dog has this weight")
        self._weight = weight

    weight: float = property(lambda self: self._weight)

cat.py: id. dog

init.py: imports the required submodules and initializes animal.subclasses

from .animal import Animal
from .dog import Dog
from .cat import Cat
from . import animal as _animal    # the initial _ makes the variable protected

_animal.subclasses = [Dog, Cat]

From that point, the documented interface only contains the zoo package itself and its classes Animal, Dog and Cat.

It can be used that way:

from zoo import Animal, Dog, Cat

if __name__ == "__main__":

    a1 = Dog(34)
    try:
        a2 = Dog(0.9)  # ValueError
    except ValueError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

    a3 = Cat(0.8)
    try:
        a4 = Cat(25)  # ValueError
    except ValueError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

    a5 = Animal(80)  # can only be dog; should return dog.
    assert type(a5) is Dog
    a6 = Animal(0.7)  # can only be cat; should return cat.
    assert type(a6) is Cat
    a7 = Animal(10)  # can be both; should return dog.
    assert type(a7) is Dog
    try:
        a8 = Animal(400)
    except NotImplementedError:
        pass
    else:
        raise RuntimeError("Should have raised Exception!")

This structure allows simple direct dependencies. It could even be improved to allow optional subclasses that could be added by a specific function declared (or imported) in __init__.py

Thanks @SergeBallesta, using `__init__.py` is another way of getting the references into `animal.py`, that I hadn't thought about. I don't like that `__init__.py` contains code that alters the modules like that - I usually think of `__init__.py` to pull/import whatever it needs, without side effects, and would not stop to think it might have some. But maybe that's just me; in classes `__init__` does do a lot more as well, after all — ElRudi, Feb 01 '22 at 16:03

Avoiding circular imports when base class returns a subclass instance in an importable module

Summary

Starting point

Refactor into importable module, in separate files

Possible solutions

A: Import modules and move to after Animal class definition

B: Move imports inside function

C: Remove imports and find class by name

D: Update dummy variable

E: a better solution??

1 Answers1

A: Import modules and move to after `Animal` class definition