0

Would it be possible to create a python module that lazily downloads and installs submodules as needed? I've worked with "subclassed" modules that mimic real modules, but I've never tried to do so with downloads involved. Is there a guaranteed directory that I can download source code and data to, that the module would then be able to use on subsequent runs?

To make this more concrete, here is the ideal behavior:

  • User runs pip install magic_module and the lightweight magic_module is installed to their system.
  • User runs the code import magic_module.alpha
  • The code goes to a predetermine URL, is told that there is an "alpha" subpackage, and is then given the URLs of alpha.py and alpha.csv files.
  • The system downloads these files to somewhere that it knows about, and then loads the alpha module.
  • On subsequent runs, the user is able to take advantage of the downloaded files to skip the server trip.
  • At some point down the road, the user could run a import magic_module.alpha ; alpha._upgrade() function from the command line to clear the cache and get the latest version.

Is this possible? Is this reasonable? What kinds of problems will I run into with permissions?

Community
  • 1
  • 1
Austin Cory Bart
  • 2,159
  • 2
  • 19
  • 32

1 Answers1

0

Doable, certainly. The core feature will probably be import hooks. The relevant module would be importlib in python 3.

Extending the import mechanism is needed when you want to load modules that are stored in a non-standard way. Examples include [...] modules that are loaded from a database over a network.

Convenient, probably not. The import machinery is one of the parts of python that has seen several changes over releases. It's undergoing a full refactoring right now, with most of the existing things being deprecated.

Reasonable, well it's up to you. Here are some caveats I can think of:

  • Tricky to get right, especially if you have to support several python versions.
  • What about error handling? Should application be prepared for import to fail in normal circumstances? Should they degrade gracefully? Or just crash and spew a traceback?
  • Security? Basically you're downloading code from someplace, how do you ensure the connection is not being hijacked?
  • How about versionning? If you update some of the remote modules, how can make the application download the correct version?
  • Dependencies? Pushing of security updates? Permissions management?

Summing it up, you'll have to solve most of the issues of a package manager, along with securing downloads and permissions issues of course. All those issues are tricky to begin with, easy to get wrong with dire consequences.

So with all that in mind, it really comes down to how much resources you deem worth investing into that, and what value that adds over a regular use of readily available tools such as pip.

(the permission question cannot really be answered until you come up with a design for your package manager)

spectras
  • 13,105
  • 2
  • 31
  • 53