How to customize python installation in both normal and develop mode? (Why `setuptools.command.develop` is not called in develop/editable mode?)

Question

I am building a example_package on which I would like to customize the installing process in both normal and develop mode. Here is the project structure:

.
├── cpp
│   ├── CMakeLists.txt
│   └── helloworld.cpp
├── pyproject.toml
├── setup.py
└── src
    └── example_package
        └── __init__.py

The content of each file is shown below

• pyproject.toml: A minimum information is added.

[build-system]
requires = ["setuptools>=62.0.0", "wheel", "cmake>=3.20"]
build-backend = "setuptools.build_meta"

[project]
name = "example_package"
version = "0.0.1"
requires-python = ">=3.8"

[tool.setuptools.packages.find]
where = ["src"]

• setup.py:

from setuptools import setup
from setuptools.command import develop, build_py, install
import subprocess

class CustomDevelop(develop.develop):
    def run(self):
        super(CustomDevelop, self).run()
        print("Execute CustomDevelop ......")

class CustomBuildPy(build_py.build_py):
    def run(self):
        super(CustomBuildPy, self).run()
        print("Execute CustomBuildPy ......")

class CustomInstall(install.install):
    def run(self):
        super(CustomInstall, self).run()
        print("Execute CustomInstall ......")
        
setup(
    cmdclass={
        'develop': CustomDevelop,
        'build_py': CustomBuildPy,
        'install': CustomInstall
    }
)

Installation

Normal mode

• Create, activate venv, and update pip and setuptools:

  python3 -m venv venv/ && source venv/bin/activate && python3 -m pip install --upgrade pip setuptools
  

• Install in normal mode:

  python3 -m pip install -vv . 2>&1 | grep -E "running|Execute"
  

• The output message:

  running egg_info
  running dist_info
  running bdist_wheel
  running build
  running build_py
  running egg_info
  Execute CustomBuildPy ......
  running install
  running install_lib
  running install_egg_info
  running install_scripts
  Execute CustomInstall ......

which is expected. CustomBuildPy and CustomInstall were executed.

Develop mode

• Create, activate venv, and update pip and setuptools:

  python3 -m venv venv/ && source venv/bin/activate && python3 -m pip install --upgrade pip setuptools
  

• Install in develop mode:

  python3 -m pip install -vv -e . 2>&1 | grep -E "running|Execute"
  

• The output message:

  running egg_info
  running dist_info
  running editable_wheel
  running build_py
  Execute CustomBuildPy ......
  running egg_info

which is not expected by me, because only CustomBuildPy is run but not CustomDevelop.

What I really want to do

The reason that I need to execute the costumed command in the installing process in the normal and develop mode is that I have a helloworld.cpp which needs to be compiled into a executable helloworld and put into the sys.path.

• cpp/helloworld.cpp: For example, the content of helloworld.cpp is trivial:

#include <iostream>
int main(int argc, char **argv) {
  std::cout << "Hello World!!!\n";
  return 0;
}

• cpp/CMakeLists.txt: the associated CMakeLists.txt is also trivial:

cmake_minimum_required(VERSION 3.20)
project(HelloWorld)
add_executable(helloworld helloworld.cpp)

The executable helloworld can be run in the module example_package like below:

$ cat src/example_package/__init__.py 
import subprocess, os, sysconfig, sys
src_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
os.environ['PATH'] = src_dir + os.pathsep + os.environ['PATH']
subprocess.run("helloworld", env=os.environ)

Lastly, I change the CustomBuildPy to:

class CustomBuildPy(build_py.build_py):
    def run(self):
        super(CustomBuildPy, self).run()
        print("Execute CustomBuildPy ......")
        subprocess.check_call("cmake -S cpp/ -B cpp/cmakebuild && cmake --build cpp/cmakebuild", shell=True)
        subprocess.check_call("cp cpp/cmakebuild/helloworld build/lib/.", shell=True)

When installing in normal mode, everything works great. The executable helloworld appears in venv/lib/python3.8/site-packages. But when installing in develop mode, the error emerges:

  subprocess.CalledProcessError: Command 'cp -r cpp/build/helloworld build/lib/.' returned non-zero exit status 1.

Because the build folder is not created in the develop mode in the source folder ..

What can I do to fix it?

Answer 1

I still can not figure out why setuptools.command.develop is not called in develop mode. But I find out the solution to achieve what I want to do: make the executable helloworld appear in the sys.path correctly in both normal and develop mode.
The key is to maintain the relative path between the executable helloworld and the package example_package. So, the module in example_package can always add the relative path between helloworld and example_package into os.environ['PATH'] in both normal and develop mode.

Minimal reproducible example

Firstly, change CustomBuildPy to the following:

class CustomBuildPy(build_py.build_py):
    def run(self):
        super(CustomBuildPy, self).run()
        print("Execute CustomBuildPy ......")
        subprocess.check_call("cmake -S cpp/ -B cpp/cmakebuild && cmake --build cpp/cmakebuild", shell=True)

        # Copying `helloworld` to `src/.` doesn't make sense in normal mode, but it doesn't hurt.
        subprocess.check_call("cp cpp/cmakebuild/helloworld src/.", shell=True)
        
        # Copying `helloworld` to `build/lib/.` makes `helloworld` appear in venv/lib/python3.8/site-packages
        # in normal mode. This line doesn't work in develop mode. But again, it doesn't hurt.
        subprocess.check_call("cp cpp/cmakebuild/helloworld build/lib/.", shell=True)

• subprocess.check_call("cp cpp/cmakebuild/helloworld src/.", shell=True) works in develop mode.
• subprocess.check_call("cp cpp/cmakebuild/helloworld build/lib/.", shell=True) works in normal mode.

Pip Install in normal mode

• Create, activate venv, and update pip and setuptools:

  python3 -m venv venv/ && source venv/bin/activate && python3 -m pip install --upgrade pip setuptools
  

• Install in normal mode:

  python3 -m pip install .
  

• Verify installation:

>>> import example_package
Hello World!!!

Pip Install in develop mode

• Create, activate venv, and update pip and setuptools:

  python3 -m venv venv/ && source venv/bin/activate && python3 -m pip install --upgrade pip setuptools
  

• Install in develop mode:

  python3 -m pip install -e .
  

• Verify installation:

>>> import example_package
Hello World!!!