What is the right way to ship dynamic library *.so with python package?

Question

I have a package require a.so which load b.so and load it by ctypes.cdll.LoadLibrary(a_so_path).

So I have to ship them with my package.

Here are two idea for it.

set LD_LIBRARY_PATH to my site-package

Now the problem is, how can I change lib load path in the runtime, set LD_LIBRARY_PATH in my module will not work(refer to Set LD_LIBRARY_PATH before importing in python). But it is a package, I can not make every user load python in a shell script which set LD_LIBRARY_PATH to ***/lib/python3.8/site-packages/mypackage/lib/.

install *.so to system lib

I found data_files in setup.cfg can distribute files out of site-packages ( to /usr/lib where can be find). But unfortunately, it not support egg or wheel package. see https://packaging.python.org/guides/distributing-packages-using-setuptools/#id55

So, what is the right way to release my package?

I believe you want to build a [wheel](https://pythonwheels.com). — chepner, Aug 05 '21 at 10:30
@chepner, I have build a wheel, but where should I place the so files? Root directory of my package? — PaleNeutron, Aug 05 '21 at 10:32
I was under the impression that the shared libraries were included *in* the wheel, and that `pip` takes care of installing the wheel properly for you. — chepner, Aug 05 '21 at 10:34
Or are you asking where they should be installed within the wheel in the first place? — chepner, Aug 05 '21 at 10:35
You can include `package_data` option for `setup` function to specify the files you want to ship with. https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html#data-files-support. The you can use either `importlib.resources` or `pkg_resources` module to find the location of the files in the package. — Kota Mori, Aug 05 '21 at 10:35
I would wonder if .so files work in all platforms. Don't you have to compile them for separate environment? — Kota Mori, Aug 05 '21 at 10:40
@chepner, @Kota Mori, the problem is not I can not contain `*.so` in package but I can't load them when import. The only way I know to import so any where is set `LD_LIBRARY_PATH` before python process start. — PaleNeutron, Aug 05 '21 at 10:41

score 2 · Accepted Answer · answered Aug 13 '21 at 01:20

Both ways in question could not work. I finally choose change RUNPATH of my dynamic library.

Use patchelf ( or chrpath if your *.so already have RUNPATH), put all dynamic libraries in one folder and run:

find . -name "*.so" | xargs -l1 -t patchelf --set-rpath "\$ORIGIN"

Dynamic library will search depends in RUNPATH while "\$ORIGIN" means current binary path, so if we put all *.so in one directory, depended *.so will be found.

then package them in wheel use

from setuptools import setup, find_packages
setup(
    ...
    package_data={
        # If any package contains *.so, include them:
        "": ["*.so",],
    }
)

now, everything works.

What is the right way to ship dynamic library *.so with python package?

1 Answers1