Python component scan

Question

I have a decorator named, say, @find_me. I want to find all classes that are decorated with it.

Or, I have a class named, say, FindMe, and I want to find all subclasses of it.

Why? because I want to do something with those classes before they are being import-ed.

So I read about __subclasses()__ and about finding decorators. The problem with the solutions I found is that the class has to be import-ed before the code runs.

In other words, if I have:

in module ${proj_root}/some_path/FindMe.py a class FindMe(object):,
and in module ${proj_root}/some_other_path/NeedsToBeFound.py, a class NeedsToBeFound(FindMe):,
and another module ${proj_root}/yet_another_path/some_module.py,
and if some_module.py looks something like:

import ... FindMe

...

subclasses_of_FindMe = FindMe.__subclasses()__

then the expected class NeedsToBeFound won't be in the result (assuming there was no import of it somewhere along the way).

So I guess I'm looking for a way to do some sort of a component scan over all python classes (that are located in the subtree of ${proj_root}).

How what would it be simpler to do: find decorators or find subclasses? And how can I do that?...

Thanks in advance!

Python has no built in way to do what you are asking for. So you will have to custom write something to do that. While I don't recommend it. You can get all subclasses of a class in python. See here https://stackoverflow.com/questions/3862310/how-to-find-all-the-subclasses-of-a-class-given-its-name As for the decorators, I don't know of a way to do that. — Rashid 'Lee' Ibrahim, Mar 25 '20 at 15:31
if you want to do something with a class before it's _imported_ - you'll need to modify the code on disk. the class doesn't exist until it has been imported. if you are looking to find all classes for a module on disk, and generate some code to do something, maybe take a look at the way doc generators like sphinx walk a module and extract docstrings.... — Corley Brigman, Mar 25 '20 at 15:38
@Rashid'Lee'Ibrahim, yes, you can get subclasses in python, but those classes needs to be `import`-ed before... The link you have kindly provided refers to different classes **within the same module**; it wouldn't work if the classes are in different packages. It even says so explicitly: "*... if the subclass's module hasn't been imported yet - then that subclass doesn't exist yet, and `__subclasses__` won't find it.*" — Ohad, Mar 25 '20 at 16:30
@CorleyBrigman, yes, "_the class doesn't exist until it has been imported._" - this is exactly what I was saying... And it is an interesting lead, regarding doc generators. I'll try to find the source code for such. Thanks! — Ohad, Mar 25 '20 at 16:32

Echsecutor · Answer 1 · 2020-03-25T15:37:07.593

-1

Have you considered putting the classes/modules for which you "want to do something before they are imported" into a common package and use __init__.py? That's what this file is actually for. ;)

If all packages are in one module, you can also just put the initializer code into that module (outside any def).

See also e.g.

edited Mar 25 '20 at 15:37

answered Mar 25 '20 at 15:32

Echsecutor

775
8
11

no... I am rookie to python, but as far as I understand, `__init__.py` is being implicitly called when the package is `import`-ed... In my case, I want to do things BEFORE it is loaded... – Ohad Mar 25 '20 at 16:24

score -1 · Answer 2 · answered Apr 02 '20 at 10:07

OK, So I came up with something this:

# /{some_proj_home_dir_1}/{some_path}/.../FindMe.py
class FindMe(object):
    pass



# /{some_proj_home_dir_1}/{some_path}/.../A.py

class A(object):

    ...

    def foo(self, project_home_dir):
        # going to mess with sys.path, so preserve current length of it, later to be restored
        sys_path_length = _append_dir_to_sys_path(project_home_dir)

        try:
            # dynamically load all modules in "the other" project
            self._load_all_modules(project_home_dir)
            # find all classes that needs to be found
            all_subclasses = self._find_all_needs_to_be_found()

            # do something with all_classes
            ...

        finally:
            # restore sys.path back to what it was
            sys.path = sys.path[:sys_path_length]


    def _load_all_modules(self, project_home_dir, subpackages=None):
        subdir = project_home_dir if subpackages is None else "/".join([project_home_dir] + subpackages)
        for (module_loader, name, ispkg) in pkgutil.iter_modules([subdir]):
            if ispkg:
                self._load_all_modules(project_home_dir, [name] if subpackages is None else subpackages + [name])
                return

            module_name = ".".join(subpackages)
            importlib.import_module("." + name, module_name)


    def _find_all_needs_to_be_found():
        return [FindMe] + FindMe.__subclasses__()


def _append_dir_to_sys_path(project_dir):
    path_length = len(sys.path)
    path = os.path.abspath(project_dir)

    while _is_part_of_package(path):
        sys.path.append(path)
        path = os.path.abspath(os.path.join(path, ".."))

    sys.path.append(path)

    return path_length


def _is_part_of_package(path):
    return os.path.isfile(os.path.join(path, "__init__.py"))



# /{some_different_proj_home_dir_2}/{some_path}/.../NeedsToBeFound.py
class NeedsToBeFound(FindMe):
    pass

Now calling A().foo({some_different_proj_home_dir_2}) seems to work :)

Python component scan

2 Answers2