How to include sql files from a folder to setuptools while packaging as python egg

Question

I have a directory structure as below.

.
├── package
    ├── app
    │   ├── __init__.py
    │   ├── file1.py
    │   └── file2.py
    ├── master_sql_folder
    │   │
    │   │──sql_folder_1
    │   │      ├── world.sql
    │   │      ├── earth.sql
    │   │
    │   └──sql_folder_2
    │          ├── planet.sql
    │          ├── sun.sql
    │         
    └── wrapper_scripts
    │    ├── wrapper.py
    │    └── __init__.py
    ├── setup.py

I am trying to include all the sql files that exist in sub folders of master_sql_folder ( world.sql, earth.sql, planet.sql, sun.sql ) to setuptools while packaging as egg,I cannot use sql_folder_1 and sql_folder_2 in the path as new folders can be added in future under master_sql_folder and I need the code to read them too.I have tried adding the following lines to my setup.py but its not including the sql files in the build.

package_data={'master_sql_folder':['*']}
packages=['app', 'wrapper_scripts']

I appreciate your help in advance.

Possible duplicate of [this question](https://stackoverflow.com/q/32609248/6340496). — S3DEV, Sep 30 '20 at 08:37
The mentioned link did help me to use this data_files=[('master_sql_folder/sql_folder_1', ['master_sql_folder/sql_folder_1/world.sql'])] and this works But I cannot use sql_folder_1 in the path, Is there a way that can pick the files from all the subfolders like data_files=[('master_sql_folder', ['master_sql_folder/*/*.sql'])]. It results in error :can't copy 'master_sql_folder\*\*.sql': doesn't exist or not a regular file — Somu Sinhhaa, Sep 30 '20 at 09:21
If I remember correctly, each file has to be named individually. Or, write a function that collects the files and builds the list of files for you. — S3DEV, Sep 30 '20 at 09:36

score 1 · Answer 1 · answered Sep 19 '22 at 16:19

I tried a few iterations of this without success.

In the end what worked was the official solution: https://python-packaging.readthedocs.io/en/latest/non-code-files.html

Often packages will need to depend on files which are not .py files: e.g. images, data tables, documentation, etc. Those files need special treatment in order for setuptools to handle them correctly.

The mechanism that provides this is the MANIFEST.in file. This is relatively quite simple: MANIFEST.in is really just a list of relative file paths specifying files or globs to include.:

include README.rst
include docs/*.txt
include funniest/data.json

In order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function.

super easy to do, literally just include MANIFEST.in in the same dir as your setup.py, and as per the documentation, you’ll need to supply include_package_data=True to the setup() function.

score 0 · Accepted Answer · answered Sep 30 '20 at 10:57

Based on @S3DEV 's suggestion and accepted answer here How to add package data recursively in Python setup.py?. I was able to get a solution.

#sql_file_picker.py -- script to feed files to setup.py

import glob
class FilePicker:
    def __init__(self):
        pass
    def sql_file_picker(self):
        sql_files = []
        directories = glob.glob('master_sql_folder\\**\\')
        for directory in directories:
            files = glob.glob(directory + '*.sql')
            if len(files) != 0:
                sql_files.append((directory, files))
        return sql_files

in setup.py

from wrapper_scripts.sql_file_picker import FilePicker
from setuptools import setup
setup(
    name='XXXXXXXXX',
    version='X.X.X',
    packages=['app', 'wrapper_scripts'],
    url='',
    license='',
    author='',
    author_email='',
    description='XXXXXXXXXXXXX',
    data_files=FilePicker().sql_file_picker()
)

How to include sql files from a folder to setuptools while packaging as python egg

2 Answers2

in setup.py