How to set up data folder to be accessible from everywhere in a python project

Question

In a python project, how do I setup a project-wide "data" folder, accessible from every module? I don't have a single entry point in my program, so I cannot do something like global (dataFolderPath). I would like for every module to know where the data folder is (without hardcoding the path in every module!), so it can load and write the data it needs. I'm using python 3.5 on a mac.

Thanks!

This data folder is a python module or just a collection of files with data? — Rodrigo, Nov 12 '16 at 11:30
@rod085 It's just a collection of files with data (even though this data may be instances of python object saved with pickle.dump) — Ant, Nov 12 '16 at 11:51

woockashek · Answer 1 · 2016-11-12T11:37:46.093

0

You should rather make a module that acts as a data provider. Only this module should be interested in where your data is.

Your current approach is wrong. Whenever you decide to change the location or format of your data you will have to change every module that uses this data.

//Edit: After creating a module there you can get it's location with:

import data_module
import os

path = os.path.dirname(data_module.__file__)

Now you have a absolute path to some project subdir. From here you can point to any other dir in your project using relative path.

edited Nov 12 '16 at 11:37

answered Nov 12 '16 at 11:03

woockashek

1,588
10
25

But there are different types of data, and some modules have to load some data and some have to write some data, and so on. Having a single module with all of that seems a little confusing. The data are different in types also; if I change the format of some data, I only need to change the module that deals with that type of data. But still I want all the data in a single folder (or in subfolder of a single folder) for clarity. Don't you agree? – Ant Nov 12 '16 at 11:17
Thank you for the answer; I was aware of this possibility, but I discarded it because I would need to hardcode the relative path in each module (i.e. If I change the module location, or the data folder location inside the project structure, i would need to manually modify each file). – Ant Nov 12 '16 at 11:52
@Ant: I mean rather having a one module that you are able to import from every other place – woockashek Nov 12 '16 at 11:58
I didn't get your last comment, sorry! Could you clarify? :) – Ant Nov 12 '16 at 12:27
You said you have a multiple entry points to your project. If it's a one project I assume that there are some modules widely used, if not it's not a project but a set of separate projects. So in such common module you could have a getDataPath() method – woockashek Nov 12 '16 at 12:31

score 0 · Answer 2 · answered Nov 12 '16 at 11:36

0

This is not python specific, but if you want to share a config globally among your programs you could set up a environment variable like MYPROJECT_DATA_PATH and all your scripts check this variable before loading the data. Or you could write a config file which all your programs know the location. Or both, a environment variable with the path of the config file, where you can fine-tune it for your needs.

answered Nov 12 '16 at 11:36

Rodrigo

400
3
7

Uhm, this is interesting, even though I was hoping not to mess with the environment variables for something this trivial.. – Ant Nov 12 '16 at 11:53

score 0 · Answer 3 · edited May 23 '17 at 11:53

To get data installed relative to your Python module, you could use pkgutil.get_data() or setuptools' pkg_resources.resource_string(). This works even if your module is packaged as a zip archive.

To find out an appropriate place to put user data, you could use appdirs module for portability:

import appdirs   # $ pip install appdirs

user_data_dir = appdirs.user_data_dir("App name", "Author")

To avoid duplicating the logic of where to put data in a way that is optimal for your specific application, you could create an object that will be responsible for it. You could import it directly (in simple cases) e.g., from custom_app import config or build it during an initialization and set it as an attribute of your application object app.config or as a global function custom_app.get_config() if there is none.

To get data path, ask the config object: config.get_data_dir() where config may return values derived from config files (e.g., ~/.config/custom_app/config.yml), environment variables (CUSTOM_APP_DATA_DIR), command-line options with/without the help of pkgutil/pkg_resources/appdirs modules.

It is not the only way to get the config info but it should be a good start that works in many cases.

How to set up data folder to be accessible from everywhere in a python project

3 Answers3