16

I am building a python module with Cython that links against a DLL file. In order to succesfully import my module I need to have the DLL in the Windows search path. Otherwise, the typical error message is:

ImportError: DLL load failed: The specified module could not be found.

Is there a way to packaged the DLL directly into the produced pyd file to make the distribution easier?

One example of this is with the OpenCV distribution, where a (huge) pyd file is distributed and is the only file needed for the Python bindings to work.

toine
  • 1,946
  • 18
  • 24
  • 2
    The DLL can be in the same directory as the extension module. Python loads a .pyd extension using `LoadLibraryEx` with `LOAD_WITH_ALTERED_SEARCH_PATH`. – Eryk Sun Jul 26 '15 at 14:10
  • I am looking for a way to merge both file into 1, skipping the need to have the DLL in the search path. – toine Jul 26 '15 at 17:20
  • Not sure, but I think you might be able to use ctypes.cdll or ctypes.windll to load your library first, and then it will find the instance of it, e.g. temp = ctypes.cdll.LoadLibrary('path_to_dll'). Alternatively, from within your pyd, you could access os.environ() to update the program's PATH before trying to load the DLL. – Patrick Maupin Jul 30 '15 at 22:40
  • 3
    If you have the source for the other DLL, you could certainly create a makefile to link them together. Not sure if there are any packages that let you relink. – Patrick Maupin Jul 30 '15 at 22:59
  • That is the relink way I am looking for : ) – toine Aug 03 '15 at 20:03

1 Answers1

10

Python's packaging & deployment is still a pain point for many of us. There is just not a silver bullet. Here are several methods:

1. OpenCV build method

The method is decribed here : https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_bindings/py_bindings_basics/py_bindings_basics.html#bindings-basics

OpenCV generates these wrapper functions automatically from the C++ headers using some Python scripts which are located in modules/python/src2.

Basically it parse the header files and generate the static PyObject keywords whenever it's needed. Once the header are created appropriately, it just call python setup. Honestly, it might work, but I would not advise this method.

2. Makefiles

If you already use Makefile, just create a rule to place your lib accordinlgy. Example, from my own code :

setup.py

from distutils.core import setup, Extension
setup(name='sha1_hmac', version='1.0',  \
      ext_modules=[Extension('sha1_hmac',
                             library_dirs=['C:\MinGW\lib'],
                             sources= ['../tools/sha1.c','sha1_hmac.c'])])

Makefile

# The hmac generation used by the webserver is done
# using the sha1.c implementation. There is a binding needed to
# glue the C code with the python script
libsha1_hmac:
ifeq ($(OS), Windows_NT)
    $(PYTHON) setup.py build --compiler=mingw32
else
    $(PYTHON) setup.py install --home=$(CURDIR)
endif

.PHONY: webserver
webserver:  libsha1_hmac
ifeq ($(OS), Windows_NT)
    mv $(shell find build -type f -name "sha1*.pyd") $(LIB)
else
    mv -f $(shell find $(LIB)/python -type f -name "sha1*.so") $(LIB)
endif
    $(PYTHON) hmac_server.py

3. Modern deployement tools

There are several new tools to deploy python applications, namely wheels which seem to gain traction. I don't use it, but it look like it can ease up your bundling problem :

Once it wheeled, you can install it like this : pip install some-package.whl

Community
  • 1
  • 1
lucasg
  • 10,734
  • 4
  • 35
  • 57
  • The openCV build method is producing the 1 file pyd result I am looking for (with access to the sources). For the method 2, I don't know if you can put the winDLL as source. In my case, I have them in libraries/library_dirs. For the method number 3, it seems like the DLL will be distributed but not merged into the final pyd. – toine Aug 03 '15 at 20:00
  • Do you have access to the DLL source code ? or do you just link it, like user32.dll for example ? Only the first case works with OpenCV's method. – lucasg Aug 03 '15 at 20:34
  • I do have access to the source code in my special case, and yes rebuild everything together in a unique build project would work. However, I am looking for a solution in case there is no access to the source code. – toine Aug 03 '15 at 20:56