464

I have a command line script that I run with a lot of arguments. I have now come to a point where I have too many arguments, and I want to have some arguments in dictionary form too.

So in order to simplify things I would like to run the script with a settings file instead. I don't really know what libraries to use for the parsing of the file. What's the best practice for doing this? I could of course hammer something out myself, but if there is some library for this, I'm all ears.

A few 'demands':

  • Rather than using pickle I would like it to be a straight forward text file that can easily be read and edited.
  • I want to be able to add dictionary-like data in it, i.e., some form of nesting should be supported.

A simplified pseudo example file:

truck:
    color: blue
    brand: ford
city: new york
cabriolet:
    color: black
    engine:
        cylinders: 8
        placement: mid
    doors: 2
dreftymac
  • 31,404
  • 26
  • 119
  • 182
c00kiemonster
  • 22,241
  • 34
  • 95
  • 133
  • 8
    The particular syntax of this example file is actually YAML, check Benson's answer. – Skippy le Grand Gourou Jan 24 '17 at 11:18
  • 1
    I'd suggest using `python-box`, see this [answer](https://stackoverflow.com/a/63071957/2137370). – evolved Jul 24 '20 at 11:01
  • I recommend giving [trapdoor](https://github.com/claymcleod/trapdoor/) a try for turn-key configuration (disclaimer: I'm the author of trapdoor). – clmcleod Nov 27 '21 at 17:58
  • 2
    As of Python 3.11, the `tomlllib` package became part of standard library and is the preferred way for parsing configuration in the awesome [TOML format](https://toml.io/en/). (For earlier Python versions, the corresponding `tomli` package can be pip-installed.) TOML relies on a formal specification (unlike INI), is easy to read and allows comments (unlike JSON), and limits complexity (unlike YAML). – sammy May 14 '23 at 22:53

4 Answers4

277

You can have a regular Python module, say config.py, like this:

truck = dict(
    color = 'blue',
    brand = 'ford',
)
city = 'new york'
cabriolet = dict(
    color = 'black',
    engine = dict(
        cylinders = 8,
        placement = 'mid',
    ),
    doors = 2,
)

and use it like this:

import config
print(config.truck['color'])
oberbaum
  • 2,451
  • 7
  • 36
  • 52
dugres
  • 12,613
  • 8
  • 46
  • 51
  • 102
    This is a pretty bad idea as if you want to allow low-privileged users to be able to change configuration files only, this way you're essentially allowing them to sneak in privileged code. – nikolay Jun 27 '12 at 23:17
  • 212
    Allowing "low-privileged" users to change config for a more privileged program is probably a questionable setup anyway. – XTL Dec 12 '12 at 13:36
  • 2
    This gives no protection against "low-privileged" users changing the configuration. If you import the module at the beginning of a script, change the value of one of the variables and the import the rest of the modules you can modify the configuration values even if you don't have permission to write the configuration file. – Javier Castellanos Jan 22 '14 at 23:37
  • 2
    Perhaps it is not in the current working directory (cwd). In that case, you have to make it visible to Python by either changing the cwd with os.chdir (use os.getcwd() to know where you are) or adding the config file location to the PYTHONPATH (not recommended). Hope this helps. – Chris Apr 25 '14 at 07:06
  • Otherwise, if the config file is in a subdir, you can turn that subdir into a package by placing an empty `__init__.py` file in it. Then you could import your config file with `import subdir.config` – Chris Apr 25 '14 at 07:15
  • 26
    You may also run into issues packaging your project for deployment using a tool such as py2app. The user may not be able to edit the configuration file once it's distributed since it would invalidate the app's signature. – bschwagg Jan 17 '15 at 18:40
  • If using python as a config file, you can use exec() to load it into a dictionary or imp.new_module to turn it into a module. This way the configuration is not in the package and can be placed in a system-standard config location if you prefer (like /etc). For more advanced usage you can also prepopulate the dict you pass to exec with objects that your config file can use as a simple DSL. – Jeremy Jan 15 '16 at 17:46
  • 34
    The main disadvantage with this (otherwise very convenient option) is that ``.py`` files are executable, so any kind of code could be run while trying to load the configuration through ``import``. That's unacceptable from a security standpoint. – Apalala Feb 09 '16 at 23:18
  • 7
    Can't a version of this be done safely with `ast.literal_eval`? https://docs.python.org/3/library/ast.html#ast.literal_eval – André C. Andersen Sep 02 '16 at 21:49
  • 1
    (1) security is not always in issue, it really depends on the project. (2) A problem I found with config files like this is if you need to create them programaticlly. Then it is difficult – Jakobovski Jan 07 '19 at 13:04
  • 1
    Note, there is the [Python-like (Python dialect/subset) Starlark language](https://github.com/bazelbuild/starlark), which is intended for this use case of config files (and used by Bazel). The main feature is hermetic execution, i.e. execution cannot access the file system, network, system clock. It is safe to execute untrusted code. – Albert Mar 03 '21 at 13:00
  • 1
    Actually `import config` might not always work, see [python - How to import a module given the full path? - Stack Overflow](https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path) – user202729 Oct 24 '21 at 01:14
  • As others sort of allude to, this seems to become a bit more complex if you're having multiple directories of scripts within your package and you want them all to reference the settings file, as there are [plenty of topics about the complexities of relative imports in Python](https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time) – JeopardyTempest Jun 20 '22 at 05:39
  • @Apalala any kind of file is executable, as long as its content is understood by the interpreter. File extension has nothing to do with it. – Itération 122442 Jul 28 '22 at 06:04
  • Agree, @Itération122442. Every input file is a program in some language that the interpreter understands. Yet in the context of configuration files we prefer to use languages or language subsets that are much less than Turing-complete, so they cannot be used to write an interpreter: JSON, YAML, TOML. – Apalala Jul 29 '22 at 13:04
260

The sample config you provided is actually valid YAML. In fact, YAML meets all of your demands, is implemented in a large number of languages, and is extremely human friendly. I would highly recommend you use it. The PyYAML project provides a nice python module, that implements YAML.

To use the yaml module is extremely simple: 

import yaml
config = yaml.safe_load(open("path/to/config.yml"))
Alan W. Smith
  • 24,647
  • 4
  • 70
  • 96
Benson
  • 22,457
  • 2
  • 40
  • 49
  • 7
    yaml is always something I turn to; the format can be from dead simple to supporting embedded python code, and the standard library does the heavy lifting of parsing and sanitation for you. – Todor Minakov Oct 10 '15 at 05:56
  • 24
    Agreed. For you or users writing YAML, [here is the best YAML reference that I know of](http://camel.readthedocs.org/en/latest/yamlref.html). The [official documentation](http://yaml.org/) is unfortunately a spec aimed at implementers, and nothing else, but Eevee's guide is fantastic. – Esteis Jan 29 '16 at 13:33
  • 12
    For us uninitiated, that's ```pip3 install pyyaml``` to get it ready to import into python scripts. – user8675309 Mar 11 '19 at 04:36
  • 8
    Beware, yaml is only friendly if you keep it very simple, it by default has tons of problematic, bordering on unsafe features. Try https://hitchdev.com/strictyaml/ instead as a safe-by-default lite alternative. – Gringo Suave Oct 30 '19 at 23:45
  • 3
    See [Munch](https://pypi.org/project/munch/), https://stackoverflow.com/questions/52570869/load-yaml-as-nested-objects-instead-of-dictionary-in-python `import yaml; from munch import munchify; f = munchify(yaml.load(…)); print(fo.d.try)` – Hans Ginzel Jun 21 '20 at 20:35
  • @HansGinzel You should make an answer on its own, as what you are suggesting is way easier to use – Begoodpy Oct 29 '20 at 13:57
170

I Found this the most useful and easy to use https://wiki.python.org/moin/ConfigParserExamples

You just create a "myfile.ini" like:

[SectionOne]
Status: Single
Name: Derek
Value: Yes
Age: 30
Single: True

[SectionTwo]
FavoriteColor=Green
[SectionThree]
FamilyName: Johnson

[Others]
Route: 66

And retrieve the data like:

>>> import ConfigParser  # For Python 3 use the configparser module instead (all lowercase)
>>> Config = ConfigParser.ConfigParser()
>>> Config
<ConfigParser.ConfigParser instance at 0x00BA9B20>
>>> Config.read("myfile.ini")
['c:\\tomorrow.ini']
>>> Config.sections()
['Others', 'SectionThree', 'SectionOne', 'SectionTwo']
>>> Config.options('SectionOne')
['Status', 'Name', 'Value', 'Age', 'Single']
>>> Config.get('SectionOne', 'Status')
'Single'
Wenuka
  • 887
  • 2
  • 9
  • 25
Maviles
  • 3,209
  • 2
  • 25
  • 39
67

Yaml and Json are the simplest and most commonly used file formats to store settings/config. PyYaml can be used to parse yaml. Json is already part of python from 2.5. Yaml is a superset of Json. Json will solve most uses cases except multi line strings where escaping is required. Yaml takes care of these cases too.

>>> import json
>>> config = {'handler' : 'adminhandler.py', 'timeoutsec' : 5 }
>>> json.dump(config, open('/tmp/config.json', 'w'))
>>> json.load(open('/tmp/config.json'))   
{u'handler': u'adminhandler.py', u'timeoutsec': 5}
Anoop
  • 1,757
  • 1
  • 19
  • 24
  • 18
    While more or less equivalent, json isn't nearly as human readable as yaml. Since his sample config is actually valid yaml, I'd stress that instead of json. – Benson Feb 20 '11 at 22:23
  • 28
    Using "json.dump(config, fp, sort_keys=True, indent=4)" improves readability. – phobie Jul 01 '13 at 16:55
  • 4
    [A simple python class loading its data members from a JSON file in RAII manner](https://gist.github.com/nadya-p/b25519cf3a74d1bed86ed9b1d8c71692) – Hope May 16 '18 at 10:16