9

I have a following package to distribute:

mymodule/
    data/
        1.txt
    mymodule/
        __init__.py
    tests/
        test_mymodule.py
    setup.py

In order to install it under a vitualenv I apply this command:

pip install .

Everything is installed fine but the path to my data file becomes broken.

>>> from mymodule import get
>>> print(get())
...
FileNotFoundError: [Errno 2] No such file or directory: '/home/alexander/Stuff/pip_example/mymodule_test/venv/lib/python3.5/site-packages/mymodule/../data/1.txt'

I made a research and found, the folder data was create in the root of the virtualenv's folder that caused the error. How should I improve my code to keep my tests working and to have the correct path to the data file?

The contents of the files:

data/1.txt

yes

mymodule/__init__.py

import os

src = os.path.join(
    os.path.dirname(__file__),
    '../data/1.txt'
)

def get():
    with open(src) as f:
        return f.read().strip()

tests/test_mymodule.py

import unittest
import mymodule

class MymoduleTest(unittest.TestCase):
    def test_get(self):
        s = mymodule.get()
        self.assertEqual(s, "yes")

setup.py

from distutils.core import setup

data_files = [
    ('data', ['data/1.txt'])
]

setup(
    name='mymodule',
    version='0.0',
    packages=['mymodule'],
    data_files=data_files,
)

I am new in packaging Python modules. Please help me with this issue.

=====================================================================

I figured out that I need to use sys.prefix to access the virtualenv's root. In other words, if I edit mymodule.__init__.py this way everything will work correctly:

import os
import sys

src = os.path.join(sys.prefix, 'data/1.txt')

def get():
    with open(src) as f:
        return f.read().strip()

But after that the test fails with the error:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/data/1.txt'

This is because sys.prefix is /usr without activated virtualenv. So I need a different way to improve the package, I guess.

Fomalhaut
  • 8,590
  • 8
  • 51
  • 95
  • Minimal runnable published working example at: https://stackoverflow.com/questions/3596979/manifest-in-ignored-on-python-setup-py-install-no-data-files-installed/60735402#60735402 – Ciro Santilli OurBigBook.com Mar 18 '20 at 07:53

1 Answers1

2

  1. Check that your file is properly distributed when installing the package.

  2. sys.prefix will not locate "your" package. The __file__ attribute of the module points to the __init__.py file. You can use this for the base path as in:

    import os
import mymodule

src = os.path.join(os.dirname(mymodule.__file__), 'data/1.txt')

def get():
    with open(src) as f:
        return f.read().strip()
Pierre de Buyl
  • 7,074
  • 2
  • 16
  • 22
  • It doesn't work, because after installation the file 1.txt is located in a different place regarding the module (I mentioned it in the question). – Fomalhaut Nov 30 '17 at 18:44
  • 1
    Can you try using the `package_data` argument to `setup` (as in `package_data = {'mymodule': ['data/1.txt']}`. the dict key here is the name of the pacakge and the dict value is a list containing paths that will be installed. This should place the files within the package directory and hence in a known and findable location. – Pierre de Buyl Nov 30 '17 at 20:47
  • 1
    For this to work, `data` must be *under* `mymodule`, that is in `mymodule/mymodule/data`. The first "mymodule" just happens to be the name of the directory in which the whole project (setup file, doc, etc) is stored. Can you re-try after having moved `data` one level of `mymodule` down? – Pierre de Buyl Dec 01 '17 at 10:13
  • Thanks, it worked. But what if I want to keep my data file outside the module folder? Isn't it impossible to pack? – Fomalhaut Dec 01 '17 at 10:47
  • 2
    You can do that with distutils (see the [docs](https://docs.python.org/3.5/distutils/setupscript.html#installing-additional-files)) but setuptools is the recommended tool now and for setuptools it is [impossible](https://setuptools.readthedocs.io/en/latest/setuptools.html#non-package-data-files). – Pierre de Buyl Dec 01 '17 at 16:51