Installing test files with pyproject.toml and setuptools

Question

I'm migrating an old python project to the new pyproject.toml based system and am having trouble with getting files that are required by tests to install. Inside the pyproject.toml I have:

[tool.setuptools]
package-data = {"my_pkg_name" = ["tests/*.sdf", "tests/*.urdf", "tests/*.xml", "tests/meshes/*.obj"]}

[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"

The tests that are run with pytest require the files described under package-data. After I build and install the build, the test files are not there. How do I get those files to be installed? How to include package data with setuptools/distutils? may be related, but things have changed, and I would rather not have to create a manifest file.

The project structure looks something like:

.
├── LICENSE.txt
├── pyproject.toml
├── README.md
├── src
│   ├── my_pkg_name
│   │   ├── __init__.py
└── tests
    ├── ant.xml
    ├── humanoid.xml
    ├── __init__.py
    ├── kuka_iiwa.urdf
    ├── meshes
    │   ├── link_0.obj
    │   ├── link_1.obj
    │   ├── link_2.obj
    │   ├── link_3.obj
    │   ├── link_4.obj
    │   ├── link_5.obj
    │   ├── link_6.obj
    │   └── link_7.obj
    └── test_transform.py

The pyproject.toml has no specific package discovery related settings.

Answer 1

Looking more into this, specifically: https://setuptools.pypa.io/en/stable/userguide/datafiles.html#non-package-data-files my problem is that these files are not part of the default found packages which are just the ones under src. It is also not clear whether test files should be installed or not - many projects explicitly exclude test files from installation.

To answer the original question of how to install the test files, the solution is to make tests a package (as I already have above), then specify to find it with setup tools like so:

[tool.setuptools.packages.find]
where = ["src", "tests"]

[tool.setuptools.package-data]
"*" = ["*.sdf", "*.urdf", "*.xml", "*.obj"]

I believe package-data requires setuptools >= 61 which you can specify like:

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

Further answering the deeper issue of installed tests failing due to not being able to find test files, the solution in my github actions workflow for doing automated testing on commits was to install in editable mode. Specifically:

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install -e .
        python -m pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

This is because in my tests, I refer to resource files based on a module's path:

open(os.path.join(cfg.TEST_DIR, "simple_arm.sdf"))

in the cfg module:

import os

ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../'))
TEST_DIR = os.path.join(ROOT_DIR, 'tests')

EDIT: A better solution is to reference the test files based on the test script path rather than the package's module's path. This allows us to still test the non-editable installed project. To do this, I have instead

TEST_DIR = os.path.dirname(__file__)

in each test script, and replace references to cfg.