Importing a Python module that was not available at interpreter startup

Question

I have code which looks like this:

import sys, importlib

try:
    import pip
except ImportError:
    raise ImportError("Please install pip")

reqs = ["sh", "vcstools"]
for req in reqs:
    sys.stdout.write("checking for %s..." % req)
    try:
        importlib.import_module(req)
        print("found")
    except ImportError:
        print("missing!")
        print("Installing %s..." % req)
        pip.main(['install', '--user', req])
        #importlib.invalidate_caches() python3 only
        # XXX fails
        importlib.import_module(req)
        new_mods = True

    locs = locals()
    locs[req] = sys.modules[req]

print(sh, vcstools)

This is designed to download (rough) dependencies at runtime and import them (yes, I know, it does not respect version numbers, and I could have used virtualenv, etc. These are separate issues).

If we run this (with nothing installed in ~/.local), we get the following:

checking for sh...missing!
Installing sh...
Downloading/unpacking sh
....
Successfully installed sh
Cleaning up...
Traceback (most recent call last):
  File "test.py", line 20, in <module>
    importlib.import_module(req)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named sh

So we see that after installing the sh module, we cannot immediately import it. If we re-run the script, it will succeed, finding the sh installed in the previous run.

What I find odd is that, we do not see the same behavior for vcstools; it installs and imports just fine in the same run. What gives? Is there something special about sh?

Here is the full output from the second run. Notice that we pick up sh from the last run, we then install vcstools and import it without error:

checking for sh...found
checking for vcstools...missing!
Installing vcstools...
Downloading/unpacking vcstools
  Downloading vcstools-0.1.33.tar.gz
  Running setup.py egg_info for package vcstools

Downloading/unpacking pyyaml (from vcstools)
  Downloading PyYAML-3.11.tar.gz (248Kb): 248Kb downloaded
  Running setup.py egg_info for package pyyaml

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python2.7/site-packages (from vcstools)
Installing collected packages: vcstools, pyyaml
  Running setup.py install for vcstools

  Running setup.py install for pyyaml
    checking if libyaml is compilable
    cc -pthread -fno-strict-aliasing -O2 -pipe -DNDEBUG -O2 -pipe -fPIC -fPIC -I/usr/local/include/python2.7 -c build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c -o build/temp.openbsd-5.5-amd64-2.7/check_libyaml.o
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c: In function 'main':
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: for each function it appears in.)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)

    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Successfully installed vcstools pyyaml
Cleaning up...
(<module 'sh' (built-in)>, <module 'vcstools' from '/home/edd/.local/lib/python2.7/site-packages/vcstools/__init__.pyc'>)

This is Python-2.7 on OpenBSD.

Cheers

EDIT: Just noticed the new_mods line is redundant. I will leave it there so the line numbers in the output are not skewed.

You could try a separate class for checking/installing your modules that will call your main code (in a different file) once it's done — John Dorian, Apr 02 '14 at 18:31
@JohnDorian That is an option, although it be nice if I could use the same file. — Edd Barrett, Apr 02 '14 at 19:52
Is sh an egg? They install into their own directory and are added to sys.path on startup. Non-eggs (?!) just drop into existing directories so are available for import. — tdelaney, Apr 02 '14 at 20:32
If the only reason you want it in a single file is so that a user won't have to download multiple files, you could have it run your checking/installing code and then create a .py file with the rest of your code and then run that. It's a bit of a hack but it would work. — John Dorian, Apr 03 '14 at 01:29
I'm thinking of calling out to exec() to restart the program :\ Hacky. — Edd Barrett, Apr 03 '14 at 08:29

score 1 · Answer 1 · edited May 23 '17 at 10:28

This probably is the first package you try to install into ~/.local/... - that means that ~/.local/... is only created with pip.main(['install', '--user', req])

If ~/.local does not exist upon python startup, it is not added to sys.path, and the module will not be found (not even with importlib.invalidate_caches() ).

So you have to add that path into sys.path, or reload sys.path as indicated in this answer, then it will work...

My solution:

import pip
import importlib
import sys

def pip_import(module_name, pip_package=None):
    try:
        mod = importlib.import_module(module_name)
    except ImportError:
        pip.main(['install', '--user', pip_package if pip_package else module_name])

        # If the install path did not exist on start up it has 
        # to be added to sys.path
        if not pip.locations.user_site in sys.path:
            sys.path.append(pip.locations.user_site)

        mod = importlib.import_module(module_name)

    # Add the imported module to the global namespace so it
    # can be used just like `import module`
    globals()[module_name] = mod

Importing a Python module that was not available at interpreter startup

1 Answers1