Load "whl" module dynamically

Question

I want to be able to load modules dynamically from a *.whl files in my app. I'm using sys.path.append('module.whl') for that and it works in most cases, I can't get it to work if the module has an *.so file in it, it cannot find it locally, for example using bcrypt module, for eample:

import sys
sys.path.append("bcrypt-3.1.7-cp27-cp27m-macosx_10_6_intel.whl")
import bcrypt

I'm getting:

Traceback (most recent call last):
  File "/Users/arossert/tests/whlimport/app.py", line 6, in <module>
    import bcrypt
  File "bcrypt-3.1.7-cp27-cp27m-macosx_10_6_intel.whl/bcrypt/__init__.py", line 25, in <module>
    
ImportError: cannot import name _bcrypt

Inside the *.whl there is a _bcrypt.so file but cannot find it, is there a way around it?

I have the `bcrypt-3.1.7-cp27-cp27m-macosx_10_6_intel.whl` file, I don't want to install it but I want to "import" the module from the file, without installing. — Amir Rossert, Feb 05 '21 at 13:19
(same question asked in https://stackoverflow.com/questions/36459915/is-it-possible-to-use-a-module-without-installing-it-on-your-computer ; https://stackoverflow.com/questions/9059699/python-use-a-library-locally-instead-of-installing-it but does not address `.so` files.) — user202729, Feb 05 '21 at 13:23
Although programmatically create a virtual environment and install the wheel in it doesn't sound that bad, and is easier to work with. — user202729, Feb 05 '21 at 13:24
I can't use a virtual environment in my case, I need to be able to load modules at runtime without installing. As I mentioned, manipulating the `sys.path` works for most cases but not when it contains shared libraries (`.so`) — Amir Rossert, Feb 05 '21 at 13:25
A whl is an archive file so in theory you can unzp it and use the files in there `from zipfile import ZipFile path = 'bcrypt-3.1.7-cp27-cp27m-macosx_10_6_intel.whl' print(ZipFile(path).namelist())` — Andi Domi, Feb 05 '21 at 13:25
Even then, it's still possible to create the virtual environment then rerun itself from inside the environment. — user202729, Feb 05 '21 at 13:26
as I said, creating a virtual env is not an option in my case. @AndiDomi extracting the *.whl and loading the directory seems to work but I wonder is there is a way making it work without extracting, it will make my life much easier. — Amir Rossert, Feb 05 '21 at 13:33
can you update the code with the zip method? I will create an answer and i will expand on that — Andi Domi, Feb 05 '21 at 13:39

Maurice Meyer · Accepted Answer · 2021-02-05T13:51:33.997

1

Extract the wheel yourself:

import zipfile
import sys

whlPath = '/tmp/bcrypt-3.2.0-cp36-abi3-macosx_10_9_x86_64.whl'
targetDir = '/tmp'

with zipfile.ZipFile(whlPath, "r") as whl:
    whl.extractall(targetDir)

sys.path.append(os.path.join(targetDir, 'bcrypt'))
import bcrypt
print(bcrypt)

Out:

python /tmp/script.py
<module 'bcrypt' from '/tmp/bcrypt/__init__.py'>

Note: Make sure to have all dependencies installed!

edited Feb 05 '21 at 13:51

answered Feb 05 '21 at 13:41

Maurice Meyer

17,279
4
30
47

This does work, any explanation why it is not working without extracting? I need it in a large scale and loading the modules directly from the whl will be much easier. – Amir Rossert Feb 05 '21 at 13:53
In `bcrypt/__init__.py` there is a relative import to _bcrypt: `from . import _bcrypt` which fails (probably: because `bcypt` is not installed as a module). – Maurice Meyer Feb 05 '21 at 14:00
as far as I know shared libraries (.so files) need to be directly on a file system to be loadable. in fact the must even be within the shared library search path (environment variable LD_LIBRARY_PATH) or be imported with an absolute path. You might try to search whether there is a way to load .so files that are in a zip file without extracting them. This is not python specific, but more OS (Linux?) related – gelonida Feb 05 '21 at 14:07
@Amir Rossert: Please update your question and describe your scenario briefly. Fact: There is a relative import in `bcypt`, without changing `bcypt` sources there is no way arround! – Maurice Meyer Feb 05 '21 at 14:19
So the issue is only with the relative import? can we "change" the CWD while importing the module so it can find it? – Amir Rossert Feb 05 '21 at 14:20
If you dont like the unzipping part, how it would be possible to change the CWD **into** the `whl` file ? – Maurice Meyer Feb 05 '21 at 14:27

score 0 · Answer 2 · answered Feb 05 '21 at 13:40

0

One solution would be to unzip the file (as a whl is only an archived file) and then import it like this:

from zipfile import ZipFile 
path = 'bcrypt-3.1.7-cp27-cp27m-macosx_10_6_intel.whl' 
print(ZipFile(path).namelist())

answered Feb 05 '21 at 13:40

Andi Domi

731
2
19
48

Load "whl" module dynamically

2 Answers2