3

There is code in a proprietary library (e.g. below, called foo.bar) that I use which loads modules at runtime like this:

module_name = f"foo.bar.{some_module}"
module = importlib.import_module(module_name)

The user supplies the name of a module in some text config and that module is loaded by the library.

I want to supply some modules which add additional functionality, so I'd like to write a separate library to add further modules to the foo.bar package and bundle these up for distribution via an internal PyPi.

So someone would run pip install foo mylibrary and this would install the foo and the mylibrary libraries, but I want the classes inside mylibrary to also appear inside the foo.bar namespace.

Is this possible please?

martineau
  • 119,623
  • 25
  • 170
  • 301
John
  • 10,837
  • 17
  • 78
  • 141
  • Well, inside `foo/bar.py` you can certainly do `from mylibrary import *`. You would have to make `mylibrary` a dependency of `foo`. In other words, you can't do this without modifying the `foo` package. – Tim Roberts Dec 16 '21 at 18:43
  • @TimRoberts That's incorrect, if `foo.bar` is already designed as a [namespace package](https://www.python.org/dev/peps/pep-0420/) then this is certainly possible without modifying `foo`. – wim Dec 16 '21 at 19:05

2 Answers2

3

You're describing the basic usage of namespace packaging.

If foo.bar is already a namespace package, then contribution of new submodules by 3rd-party packages is supported.

# in setup.py
from setuptools import setup

setup(
    name="mylibrary",
    ...
    namespace_packages=["foo.bar"]
)

Installing mylibrary is able to lay down new modules within the foo.bar namespace. Follow the guide on packaging namespace packages for more detailed information.

In your case, it does seems likely foo could already be using namespace packaging, since the dynamic importlib loading shown in the question suggests that it's intended to work with arbitrarily named "some_module" submodules, perhaps provided by 3rd party "plugins".

If the proprietary library foo is not already a namespace package, and it can't be converted into one, then this isn't possible† recommended - a regular package can not should not inject modules into another package's path at install time.





†Well, actually, it is possible. For example, installing the package zen-of-numpy will make available import numpy.this or from numpy import this, even though numpy itself is not a namespace package and doesn't provide a this submodule.

wim
  • 338,267
  • 99
  • 616
  • 750
1

As @wim points out, you often can't do this at install time, however, you can do this at runtime (though I would never recommend doing so because it will cause confusion and you will eventually regret it, it can be convenient for testing)

This is called "Monkey Patching" (see What is monkey patching?)

# /usr/bin/env python3
# source from mylib.py
import foo

def baz():
    ...

foo.bar = baz

Just import both packages at least once and the import of your custom library will modify foo

import foo
import mylib

However, I again do not recommend doing this for anything other than testing ..

  • What if foo updates and adds a function with the same name as one you're patching in? Now you'll clobber it and may break the updated foo!
  • What if you want to publicly release your packages?
  • What if a developer submits bugs in your package to the upstream foo?
    ...
ti7
  • 16,375
  • 6
  • 40
  • 68