Mypy Cannot Find All Modules in Python3 Django App

Question

Background

I work in a large Python3 repo that is type checked with Mypy. In accordance with PEP420 we do not use __init__.py files.

Mypy recently added support for implicit namespace packages, but the implementation now follows the chain of import statements from a given entrypoint. In the case of this repo, (a Django app) there are many modules that are imported dynamically (like middleware) or not imported at all (tests and one-off scripts).

Consider this example:

my-py3-repo/
├── hello/
│   ├── services/
│   │   └── hello_service.py
│   └── hello.py
├── scripts/
│   ├── db/
│   │   └── migrate.py
│   └── manage.py
└── tests/
    └── hello/
        ├── services/
        │   └── hello_service_test.py
        └── hello_test.py

Assuming that hello.py imports hello_service.py, everything under the hello namespace will be type checked as expected with mypy ./hello.

Problem

However test discovery with pytest, nose, django et al works differently and hello_test.py would not usually import hello_service_test.py. There is currently no way for Mypy to discover hello_service_test.py with mypy ./tests (if not using __init__.py).

Similarly, everything under the scripts directory would suffer the same problem.

Question

How can I configure Mypy such that the scripts and tests directories are always type checked?

Configuration

# Pipfile
[dev-packages]
mypy = "==0.670"
[requires]
python_version = "3.7"

# setup.cfg
[mypy]
python_version = 3.7
ignore_missing_imports = True
namespace_packages = True

See also this issue I created in the Mypy repo.

jtschoonhoven · Answer 1 · 2019-02-13T21:19:18.330

This can't be solved in Mypy alone, but here are some useful workarounds:

1. Loop Over Files in Bash

Simple version:

find . -iname '*.py' ! -name '__init__.py' | xargs mypy

But this fails if any two files have the same name (because Mypy attempts to import them both as top-level modules). Depending on your directory structure you can solve this by looping over separate applications/services that share module names:

for dir in ./myapp/*/;
    do find ${dir%*/} -iname '*.py' ! -name '__init__.py'\
    | xargs pipenv run mypy;
done

adapted from this answer

...of course if you need this to exit with an error code, you have to wire it up yourself:

{
EXIT_STATUS=0
for dir in ./myapp/*/;
    do find ${dir%*/} -iname '*.py' ! -name '__init__.py'\
    | xargs pipenv run mypy;
    if [ $$? -eq 1 ]; then EXIT_STATUS=1; fi; \
done;
exit $EXIT_STATUS;
}

2. Just Use Init Files

Just give up and put __init__.py files next to any modules that Mypy can't find through the import chain.

I have submitted a feature request to the Mypy repo requesting a --recursive option (or similar) that would instruct Mypy to type check all .py files under a given directory and its subdirectories.

Hopefully this answer will be updated with a better solution when available.

Mypy Cannot Find All Modules in Python3 Django App

1 Answers1