Add folders and subfolders outside of main module using setuptools(setup.py)

Question

So it is yet another similar looking but different question than setuptools: adding additional files outside package and Including non-Python files with setup.py. I have structure very similar to that of first question

-module
   -python_file1.py
   -python_file2.py
-folder
   -subfolder1
      -data_file_1.txt
   -subfolder2
      -data_file_2.txt

What I want: I want to install the packages along with the folder,subfolders and files within them.

What I tried:

• Approach_1: If I move the folder inside the module then I can easily use package_data option like

package_data = {'':['folder/**/*']} but this will force me to change the structure to a more messy one. Imagine I have 10-15 sub folders.

• Approach_2: By using data_files option I could list all the files including folders/sub folders and file by simply scanning the entire repo using glob.glob('my_repo') but since I have no control over(or maybe I'm not aware of one) the target directory depending on different OS so I am unable to move the files in the correct target dir.

I'm looking for an elegant solution for either of the approaches. Sample setup.py file just for reference:

from setuptools import setup, find_packages

setup(
    name='your_project_name',
    version='0.1',
    description='A description.',
    packages=find_packages(),
    package_data={'': ['folders/**/*'},
    include_package_data=True,
    install_requires=[],
)

Answer 1

I believe something like the following could get you there:

.
├── data
│   ├── file.txt
│   └── foo
│       ├── bar
│       │   └── file.txt
│       └── file.txt
├── MANIFEST.in
├── setup.cfg
├── setup.py
└── src
    └── thing
        ├── __init__.py
        └── nothing
            └── __init__.py

MANIFEST.in

# ...
recursive-include data *.txt

setup.py

#!/usr/bin/env python3

import pathlib
import setuptools

def main():
    data_packages = [
        'thing/{}'.format(p).replace('/', '.')
        for p
        in pathlib.Path('data').glob('**')
    ]
    packages = setuptools.find_packages(where='src')
    package_dir = {
        'thing': 'src/thing',
        'thing.data': 'data'
    }

    setuptools.setup(
        # see 'setup.cfg'
        packages=packages + data_packages,
        package_dir=package_dir,
        include_package_data=True,
    )

if __name__ == '__main__':
    main()

The *.txt are packaged as package data under the thing.data package of the Python project. The data can be read for example like this:

python3 -c "import pkgutil; print(pkgutil.get_data('thing', 'data/foo/bar/file.txt').decode())"