37

How do I package a Python module together with a precompiled .so library? Specifically, how do I write setup.py so that when I do this in Python

>>> import top_secret_wrapper

It can easily find top_secret.so without having to set LD_LIBRARY_PATH?

In my module development environment, I have the following file structure:

.
├── top_secret_wrapper
│   ├── top_secret.so
│   └── __init__.py
└── setup.py

Inside __init__.py, I have something like:

import top_secret

Here's my setup.py

from setuptools import setup, Extension

setup(
    name = 'top_secret_wrapper',
    version = '0.1',
    description = 'A Python wrapper for a top secret algorithm',
    url = None,
    author = 'James Bond',
    author_email = 'James.Bond.007@mi6.org',
    license = 'Spy Game License',
    zip_safe = True,
)

I'm sure my setup.py is lacking a setting where I specify the location of top_secret.so, though I'm not sure how to do that.

Kit
  • 30,365
  • 39
  • 105
  • 149

4 Answers4

14

What I ended up doing is: 

setup(
    name='py_my_lib',
    version=version,  # specified elsewhere
    packages=[''],
    package_dir={'': '.'},
    package_data={'': ['py_my_lib.so']},
)

This way I get to import the lib by its name, and don't have another level of nestedness:

import py_my_lib

and not

from py_my_lib_wrapper import py_my_lib
Ibolit
  • 9,218
  • 7
  • 52
  • 96
  • I am not sure this is a good solution as at installation it will put the .so directly at pythonx.x/site-packages directory level. IMO a clean package distribution should put every packages files in the package directory i.e.: pythonx.x/site-packages/top_secret/ . – souch Jun 08 '23 at 12:56
3

If that library should also be compiled during install you can describe this as an extension module. If you just want to ship it add it as package_data

renefritze
  • 2,167
  • 14
  • 25
3

As is mentioned in setupscript.html#installing-package-data:

setup(
    ...
    package_data={'top_secret_wrapper': ['top_secret.so']},
)
Yan QiDong
  • 3,696
  • 1
  • 24
  • 25
1

I managed to bundle a .so (that has other .so dependancies) in its python package directory like this:

  • build the mypackage_bindings.cpython-310-x86_64-linux-gnu.so containing Python bindings of C++ using pybind11 and cmake.
  • using this minimal python package dir infrastructure:
setup.cfg
setup.py
README.md
mypackage/__init__.py
mypackage/mypackage_bindings.cpython-310-x86_64-linux-gnu.so
mypackage/some_deps.so
  • set rpath of mypackage_bindings.so and its .so dependancies to $ORIGIN using these commands on linux (so that the linker will search the deps in the same .so dir):
patchelf --set-rpath '$ORIGIN' mypackage_bindings.cpython-310-x86_64-linux-gnu.so
patchelf --set-rpath '$ORIGIN' some_deps.so
  • put in mypackage/__init__.py:
import os
import sys

cur_file_dir = os.path.dirname(os.path.realpath(__file__))

# add current file directory so that mypackage_bindings.so is found by python
sys.path.append(cur_file_dir)

# set current file directory as working dir so that mypackage_bindings.so dependancies
# will be found by the linker (mypackage_bindings.so and its deps RPATH are set to $ORIGIN)
os.chdir(cur_file_dir)

# load every symbols of mypackage_bindings into upper mypackage module
from mypackage_bindings import *
  • put in setup.py:
from setuptools import setup

setup(
    name='mypackage',
    packages=['mypackage'],
    package_dir={'mypackage': 'mypackage'},
    package_data={'mypackage': ['*.so', 'lib*']},
    description='Provides mypackage to python users',
    version='0.1',
    url='https://yo.com',
    author='truc muche',
    author_email='contact@truc-muche.com',
    keywords=['pip', 'mypackage']
    )
  • from the minimal python package dir, launch this command to create the python package:

python3 setup.py sdist

That way, there is no need to set LD_LIBRARY_PATH variable, the .so are installed in the pythonX.X/site-packages/mypackage/ directory.

souch
  • 181
  • 2
  • 3