Use "extern __declspec(dllimport)" in Cython

Question

Can I use extern __declspec(dllimport) in Cython? I am trying to wrap embree in Windows, but am not sure I can dynamically link in Cython.

I read this SO post which is great for changing C/C++ and header files directly, but I'm not sure how to implement this in a .pxd file.

For example, the Embree 2.17.7 x64 header rtcore.h defines RTCORE_API as

#ifndef RTCORE_API
#if defined(_WIN32) && !defined(EMBREE_STATIC_LIB)
#    define RTCORE_API extern "C" __declspec(dllimport)
#else
#    define RTCORE_API extern "C"
#endif
#endif

However, these are left off the function signatures that use them in the pyembree pxd file rtcore.pxd. This seems consistent with the Cython docs, which state to

Leave out any platform-specific extensions to C declarations such as __declspec()

However, even if I point the pyembree setup.py file to my downloaded embree DLL by changing the line

ext.libraries = ["embree"]

to

ext.libraries = [""C:/Program Files/Intel/Embree v2.17.7 x64/bin/embree""]

I still get 3 linking errors:

mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcMapBuffer
mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcNewTriangleMesh
mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcUnmapBuffer
build\lib.win-amd64-3.8\pyembree\mesh_construction.cp38-win_amd64.pyd : fatal error LNK1120: 3 unresolved externals

I know from this SO post and the Microsoft docs that __imp_ related linker errors are due to not finding DLLs. However, you can see in rtcore_geometry.h that it is defined:

and in rtcore_geometry.pxd it is defined:

the only difference being that the .pxd file does not include RTCORE_API in the signature.

Does anyone know how I can resolve this issue so pyembree will build?

EDIT: It should also be noted, I have added

# distutils: language=c++

to all my .pyx and .pxd files. This SO post was also reviewed, but it did not solve my problem.

UPDATE: Adding the embree.lib file to my local pyembree/embree2 folder and updating setup.py to

ext.libraries = ["pyembree/embree2/*"]

permits the code to compile via

py setup.py build_ext -i

However, the packages do not load:

>>> import pyembree
>>> from pyembree import rtcore_scene
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing rtcore_scene: The specified module could not be found.

Do I need to define the "subpackages" in my setup.py? This is my current setup.py:

from setuptools import find_packages, setup

import numpy as np
from Cython.Build import cythonize
from Cython.Distutils import build_ext

include_path = [np.get_include()]

ext_modules = cythonize(
    'pyembree/*.pyx',
    language_level=3,
    include_path=include_path)

for ext in ext_modules:
    ext.include_dirs = include_path
    ext.libraries = [
        "pyembree/embree2/*",
    ]

setup(
    name="pyembree",
    version='0.1.6',
    cmdclass = {"build_ext": build_ext},
    ext_modules=ext_modules,
    zip_safe=False,
    packages=find_packages(),
    include_package_data = True
)

and the directory structure is as follows (pyembree is the top-level folder in my .venv\lib\site-packages folder of my project):

pyembree
│   .authors.yml
│   .gitignore
│   .mailmap
│   AUTHORS
│   CHANGELOG.rst
│   LICENSE
│   MANIFEST.in
│   pyproject.toml
│   README.rst
│   setup.py
│   
├───build
│   └───temp.win-amd64-3.8
│       └───Release
│           └───pyembree
│                   mesh_construction.cp38-win_amd64.exp
│                   mesh_construction.cp38-win_amd64.lib
│                   mesh_construction.obj
│                   rtcore.cp38-win_amd64.exp
│                   rtcore.cp38-win_amd64.lib
│                   rtcore.obj
│                   rtcore_scene.cp38-win_amd64.exp
│                   rtcore_scene.cp38-win_amd64.lib
│                   rtcore_scene.obj
│                   triangles.cp38-win_amd64.exp
│                   triangles.cp38-win_amd64.lib
│                   triangles.obj
│
├───pyembree
│   │   mesh_construction.cp38-win_amd64.pyd
│   │   mesh_construction.cpp
│   │   mesh_construction.h
│   │   mesh_construction.pyx
│   │   rtcore.cp38-win_amd64.pyd
│   │   rtcore.cpp
│   │   rtcore.pxd
│   │   rtcore.pyx
│   │   rtcore_geometry.pxd
│   │   rtcore_geometry_user.pxd
│   │   rtcore_ray.pxd
│   │   rtcore_scene.cp38-win_amd64.pyd
│   │   rtcore_scene.cpp
│   │   rtcore_scene.pxd
│   │   rtcore_scene.pyx
│   │   triangles.cp38-win_amd64.pyd
│   │   triangles.cpp
│   │   triangles.pyx
│   │   __init__.pxd
│   │   __init__.py
│   │
│   ├───embree2
│   │       embree.lib
│   │       rtcore.h
│   │       rtcore.isph
│   │       rtcore_builder.h
│   │       rtcore_geometry.h
│   │       rtcore_geometry.isph
│   │       rtcore_geometry_user.h
│   │       rtcore_geometry_user.isph
│   │       rtcore_ray.h
│   │       rtcore_ray.isph
│   │       rtcore_scene.h
│   │       rtcore_scene.isph
│   │       rtcore_version.h
│   │       tbb.lib
│   │       tbbmalloc.lib
│   │
│   └───__pycache__
│           __init__.cpython-38.pyc
│
└───tests
        test_intersection.py

What's the contents of your .pxd file? You usually do `cdef extern from "header.h":`. That includes the header into your Cython-generated C file. Thus `__declspec` isn't relevant because Cython doesn't need to know about it, and it's defined correctly in the included header. — DavidW, Jul 30 '21 at 17:59
@Davidw Thanks for your help! The `pxd` file is located here: https://github.com/scopatz/pyembree/tree/master/pyembree. Let me know if there's a better way to transfer this to you. — adam.hendry, Jul 30 '21 at 18:02
I don't know the answer (sorry), but I'm pretty sure `__declspec` in Cython isn't your problem — DavidW, Jul 30 '21 at 18:12
@DavidW I actually have an update. I'll add it to the question, if you can take a look — adam.hendry, Jul 30 '21 at 18:27

adam.hendry · Accepted Answer · 2021-08-01T17:20:54.237

The code functions properly once I literally hand-copy and paste over the DLLs into the generated .egg folder in my .venv\Lib\site-packages folder:

pyembree-0.1.6-py3.8-win-amd64.egg
├───EGG-INFO
│       dependency_links.txt
│       native_libs.txt
│       not-zip-safe
│       PKG-INFO
│       SOURCES.txt
│       top_level.txt
│       
└───pyembree
    │   embree.dll
    │   freeglut.dll
    │   mesh_construction.cp38-win_amd64.pyd
    │   mesh_construction.cpp
    │   mesh_construction.py
    │   rtcore.cp38-win_amd64.pyd
    │   rtcore.cpp
    │   rtcore.py
    │   rtcore_scene.cp38-win_amd64.pyd
    │   rtcore_scene.cpp
    │   rtcore_scene.py
    │   tbb.dll
    │   tbbmalloc.dll
    │   triangles.cp38-win_amd64.pyd
    │   triangles.cpp
    │   triangles.py
    │   __init__.py
    │
    └───__pycache__
            mesh_construction.cpython-38.pyc
            rtcore.cpython-38.pyc
            rtcore_scene.cpython-38.pyc
            triangles.cpython-38.pyc
            __init__.cpython-38.pyc

However, how can I tell python to copy and paste these DLLs over? Can I put something in my setup.py file?

Edit: Per @ead's comments, the setup.py can be updated to the following to automate copying the DLLs over the the right folder (thanks @ead!):

import os
from setuptools import find_packages, setup

import numpy as np
from Cython.Build import cythonize
from Cython.Distutils import build_ext


include_path = [
    np.get_include(),
]

ext_modules = cythonize("pyembree/*.pyx", language_level=3, include_path=include_path)

for ext in ext_modules:
    ext.include_dirs = include_path
    ext.libraries = [
        "pyembree/embree2/lib/embree",
        "pyembree/embree2/lib/tbb",
        "pyembree/embree2/lib/tbbmalloc",
    ]

setup(
    name="pyembree",
    version="0.1.6",
    cmdclass={"build_ext": build_ext},
    ext_modules=ext_modules,
    zip_safe=False,
    packages=find_packages(),
    include_package_data=True,
    package_data={"pyembree": ["*.cpp", "*.dll"]},
)

That are examples how dll can be distributed: https://stackoverflow.com/a/62723124/5769463 and https://stackoverflow.com/a/63837811/5769463. You need to define 'package_data' to include dll and libs. — ead, Jul 30 '21 at 19:32
@ead I was wrong, your solution does work. I needed some time to review it. Thank you very much! — adam.hendry, Aug 01 '21 at 17:21

Use "extern __declspec(dllimport)" in Cython

1 Answers1