6

I want to build two related conda packages:

  1. A shared object file libfoo.so with compiled code
  2. A Python wrapper around that code, foopy

Upon import, the foopy module needs to locate the libfoo.so file, which it will then use with ctypes:

so_directory = ???
lib = ctypes.cdll.LoadLibrary(os.path.join(so_directory, 'libfoo.so'))

How do I reliably find where the libfoo.so file is located? I'm happy to change either recipe.

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • I suspect some of the guys you work with will know the answer to this one :-) – Mad Physicist Jan 14 '16 at 18:35
  • 5
    Indeed, and I've pointed them to this. It's nicer to ask questions and record answers in a public place though. Helps all involved rather than just me. – MRocklin Jan 14 '16 at 18:39

2 Answers2

5

I would recommend installing the .so to the PREFIX/lib folder - in other words, put it on the default search path.

For Windows/Anaconda, this is PREFIX/Library/bin instead.

EDIT:

PREFIX is just wherever you have Python installed. This might be /usr or /usr/local, or ~/miniconda

Also:

You should remove the os.path.join bit and just pass the filename into the DLL load. It will look in default paths, which as long as you are running the python from PREFIX, will include the paths mentioned above.

msarahan
  • 1,081
  • 9
  • 20
  • This package will only ever be used on Linux. What is prefix in this case? Also, can you verify that this is in user space and not in system space (where I don't necessarily have write privileges.) – MRocklin Jan 14 '16 at 18:41
  • PREFIX is just wherever you have Miniconda/Anaconda installed. It's the root of that installation. Common default example: ~/miniconda – msarahan Jan 14 '16 at 18:42
  • 1
    OK, then how can `foopy` reliably find that file? Is `~/miniconda/envs/my-env/lib` placed on the system load library path? – MRocklin Jan 14 '16 at 18:44
  • As long as you are using the Python interpreter from envs/my-env, then it will look in that folder for shared libraries. Just omit the os.path.join bit from your code sample above - it will look in default search paths for libfoo.so. – msarahan Jan 14 '16 at 18:46
  • Thanks! If you're willing to update your answer a bit with the information from the comments I'd be glad to mark it as correct. – MRocklin Jan 14 '16 at 18:51
  • 1
    For Windows, DLL names aren't versioned typically, so this approach leads to DLL hell. DLL dependencies should be kept either beside the module that loads them or in a subdirectory of the package. They can be found relative to `__file__`. If your DLL has other DLL dependencies and you need behavior similar to a Linux `RPATH` with "$ORIGIN", use `LoadLibraryEx` with `LOAD_WITH_ALTERED_SEARCH_PATH`. Or for just `ctypes.CDLL` (i.e. a plain `LoadLibrary`), use `SetDllDirectory` or extend `PATH` beforehand. – Eryk Sun Jan 17 '16 at 11:48
  • This is very true, but doesn't matter much with anaconda/miniconda, because only one version of a package can be installed in an environment at any given time. What would be very bad would be if packages all started including their own dependencies and then placing them all in this location. That would definitely be DLL hell. – msarahan Jan 17 '16 at 13:34
  • It's trivial to [add a manifest as resource #2](http://blogs.msdn.com/b/dsvc/archive/2013/03/21/how-to-load-2-dlls-with-same-name-but-different-versions.aspx) using mt.exe to an existing DLL/PYD. The activation context this creates allows each DLL to use its own version of its dependent DLLs that are listed in the manifest. Thus multiple packages can have their own "xyz.dll" dependency. Obviously they can't all dump their "xyz.dll" files in the same directory. I recommend using the package directory and getting the absolute path for the main DLL via `__file__`. – Eryk Sun Jan 17 '16 at 22:30
  • If each package has all of its own dependencies in its own folder, then you might as well use static linking and not deal with this mess at all. Manifests are good, though, and we probably will need to figure out side by side configuration or something equivalent at some point. – msarahan Jan 18 '16 at 01:12
2

I think I would use relative paths.

Here's a bit of a related post. Python ctypes: loading DLL from from a relative path

Let's say we've got a conda env activated at some random location $BAR. I'd definitely put your libfoo.so at $BAR/lib/libfoo.so. To do that, just make sure conda-build puts it at $PREFIX/lib/libfoo.so.

Then let's say we have the foopy project with the standard setup.py and code in foopy/__init__.py. That will get pip- or conda-installed to $BAR/lib/python2.7/site-packages/foopy/__init__.py. (Or python3.x) So the contents of foopy/__init__.py would be something like

import os.path
fourup = '../../../..'
libdir = os.path.normpath(os.path.join(__file__, fourup))

import ctypes
lib = ctypes.CDLL(os.path.join(libdir, 'libfoo.so')
Community
  • 1
  • 1
kalefranz
  • 4,612
  • 2
  • 27
  • 42
  • Assuming that libfoo is installed by conda this seems pretty sane to me. Presumably we could try both this and trusting the OS to find the library in order. – MRocklin Jan 15 '16 at 03:23