Trying to make a dict behave like a clean class/method structure

Question

I'm trying to make a dictionary (read from an yaml data), behave like a class. Therefore if I call for class.key I would retrieve his value. The code is listed below:

import errno
import sys
import yaml

backup_conf="""
loglevel: INFO
username: root
password: globalsecret
destdir: /dsk/bckdir/
avoidprojects: 

matchregex: /bkp/

depots:
    server1:
        password: asecret

    server2:
        username: root

    server3:

    server4:
        destdir: /disk2/bkp/

projects:
    proj1:
        matchregex: 
            - /backups/
            - /bkp/
"""

class Struct:
    def __init__(self, **entries): 
        self.__dict__.update(entries)

class Config:

    def __init__(self, filename="backup.cfg", data=None):
        self.cfg = {}
        if data is None:
            try:
                fd = open(filename,'r')
                try:
                    yamlcfg = yaml.safe_load(fd)
                except yaml.YAMLError as e:
                    sys.exit(e.errno)
                finally:
                    fd.close()
            except ( IOError, OSError ) as e:
                sys.exit(e.errno)
        else:
            try:
                yamlcfg = yaml.safe_load(data)
            except yaml.YAMLError as e:
                sys.exit(e.errno)

        self.cfg = Struct(**yamlcfg)

    def __getattribute__(self, name):
        try:
            return object.__getattribute__(self, name)
        except AttributeError:
            return self.cfg.__getattribute__(name)


    def get_depot_param(self,depot,param):
        try:
            self.depot_param = self.cfg.depots[depot][param]
        except ( TypeError, KeyError) as e:
            try:
                self.depot_param = getattr(self.cfg, param)
            except KeyError as e:
                    sys.exit(e.errno)

        return self.depot_param

    def get_project_param(self,project,param):
        try:
            self.project_param = self.cfg.projects[project][param]
        except ( TypeError, KeyError) as e:
            try:
                self.project_param = getattr(self.cfg, param)
            except KeyError as e:
                sys.exit(e.errno)

        return self.project_param

    def get_project_matches(self,project):
        try:
            self.reglist = self.cfg.projects[project]['matchregex']
        except KeyError as e:
            try:
                self.reglist = self.cfg.matchregex
            except KeyError as e:
                    print "Error in configuration file: {0}: No default regex defined. Please add a matchregex entry on conf file".format(e)
                    sys.exit(e.errno)

        if isinstance(self.reglist, str):
            self.reglist = self.reglist.split()

        return self.reglist

    def get_depots(self):
        return self.cfg.depots.keys()                                                        

if __name__ == '__main__':
    # Read config file to cfg
    config = Config(data=backup_conf)

The code runs fine and I'm able to fetch data like: config.cfg.loglevel that returns INFO as expected. But I wish to know how can I call as config.loglevel removing that cfg that cleary cames from my self.cfg instance variable. (Of course any tips to enhance the code is welcome).

It looks like you've already made an attempt using `__getattribute__`. What happens when you try to use `config.loglevel`? Do you get an exception? — mgilson, Mar 09 '16 at 17:25
Yes. I tried and didn't worked. I get: `AttributeError: Config instance has no attribute 'loglevel'` and indeed I have added that `except` inside `__getattribute__`, but still didn't worked. I'm running python 2.6.6 (environment needs it) — Lin, Mar 09 '16 at 17:28
A dictionary already **does** behave like a class. It has a [`get(keyname)` method](https://docs.python.org/2/library/stdtypes.html#mapping-types-dict) which takes a string. What you mean is, you want a dict to behave like an **object**, i.e. **allow accessing its attributes(/properties) by name without quoting**. That question has been answered many times before here, see the duplicates. — smci, Nov 13 '17 at 18:39

score 3 · Accepted Answer · edited May 23 '17 at 12:15

Well, the simplest solution would be to use PYYaml constructors, i.e. map a class to a yaml type.

① Using constructors

All you have to do is make your class a child of yaml.YAMLObject, add the yaml_tag member to tell yaml when to use that class to construct an instance of that class (instead of a dict), and you're set:

class Config(yaml.YAMLObject):
    yaml_tag = '!Config'

    @classmethod
    def load(self, filename="backup.cfg", data=None):
        self.cfg = {}
        if data is None:
            with open(filename,'r') as f:
                yamlcfg = yaml.load(f)
        else:
            yamlcfg = yaml.load(data)
        return yamlcfg

backup_conf="""
!Config
loglevel: INFO
username: root
password: globalsecret
destdir: /dsk/bckdir/
avoidprojects:

matchregex: /bkp/

depots:
    server1:
        password: asecret

    server2:
        username: root

    server3:

    server4:
        destdir: /disk2/bkp/

projects:
    proj1:
        matchregex:
            - /backups/
            - /bkp/
"""


if __name__ == '__main__':
    # Read config file to cfg
    config = Config.load(data=backup_conf)

As you might see, I'm using a factory method to load the data, and create the instances, which is what the load class method is here for.

One of the advantages of that approach, is that you can type all your elements directly by writing the type tag within your yaml data. So if you want you can also type your servers using a similar approach, making your yaml like:

depots:
   server1: !Server
     password: asecret
   server2: !Server
     username: root
   server3: !Server
   server4: !Server
     destdir: /disk2/bkp

And same way with each project within the projects key.

② Using `namedtuple`s

if you do not want to change your yaml, then you can make the Config class a child of namedtuple and when you load the yaml data, you can create the namedtuple out of a dict.

To do so, in the following snippet, I'm creating a recursive function (nested within the load class method), that walks through all dicts (and nested dicts) and convert them into namedtuples.

import yaml
from collections import namedtuple

class Config:
    @classmethod
    def load(self, filename='backup.cfg', data=None):
        """Load YAML document"""

        def convert_to_namedtuple(d):
            """Convert a dict into a namedtuple"""
            if not isinstance(d, dict):
                raise ValueError("Can only convert dicts into namedtuple")
            for k,v in d.iteritems():
                if isinstance(v, dict):
                    d[k] = convert_to_namedtuple(v)
            return namedtuple('ConfigDict', d.keys())(**d)

        if data is None:
            with open(filename, 'r') as f:
                yamlcfg = yaml.load(f)
        else:
            yamlcfg = yaml.load(data)
        return convert_to_namedtuple(yamlcfg)

and when you run it:

>>> cfg = Config.load(data=backup_conf)
>>> print cfg.username, cfg.destdir
root /dsk/bckdir/
>>> print cfg.depots.server4.destdir
/disk2/bkp/
>>> print cfg.depots.server2.username
root

③ Using a custom `yaml.Loader` to build `namedtuple`s

I tried to figure out a way to do it, but after some try and err I understood it would take me too much time to figure it out, and it would get too complex for it to be viable as an easy to understand solution. Just for the fun, here's what makes it hard to implement.

There's a way to make your own default loader, and change how the default nodes are being converted. Within the default loader, you can override the method that creates dicts to make it create namedtuples:

class ConfigLoader(yaml.Loader):
    def construct_mapping(self, node, deep=False):
        # do whatever it does per default to create a dict, i.e. call the ConfigLoader.construct_mapping() method
        mapping = super(ConfigLoader, self).construct_mapping(node, deep)
        # then convert the returned mapping into a namedtuple
        return namedtuple('ConfigDict', mapping.keys())(**mapping)

The only issue is that another method calling that one is expecting to build first the dict tree, and only then update it with values:

def construct_yaml_map(self, node):
    data = {}
    yield data ## the object is returned here, /before/ it is being populated
    value = self.construct_mapping(node)
    data.update(value)

So, as I said, there's certainly a way around, but if it's taking me too much time to figure out, there's no point to show you how to do it, as it would make it hard for you (and future readers) to understand. As I saw @user1340544's answer, you might want to consider using EasyDict instead of collections.namedtuple (if you're ok with external packages).

Conclusion

So as you can see here the data field is built as an empty dict, that dict is yielded to the caller, before the values being added to it. So then the values are only being added after the dict has been built. But the namedtuple need to be built in a single step (i.e.: you need to know all keys beforehands), so that approach cannot be used.

I personally would prefer option ①, using the tags, as you can then use the classes it maps to, to do verification of the configuration (and alert upon missing config items, or wrongly typed ones, or extra ones). You also earn from having different names for each type, making it easy to report what's wrong when parsing the configuration file, and all that with a minimum of extra code. Of course, though, option ② does the job well.

HTH

I like to read more about your ideas, but indeed, I don't want to change the YAML, neither add tags or yaml cross references (as & or *). — Lin, Mar 09 '16 at 19:29
Ok, then I'll show you the `collections.namedtuple` idea. I guess I'll come back by tomorrow on a real computer to show it off :-) The whole idea is really to just have `Config` a subclass of `namedtuple`, use the factory method to load the yaml and convert each dict into a namedtuple. — zmo, Mar 09 '16 at 19:44
Another strategy to do the same would be to use `add_constructor()` to convert all dict into `namedtuple` at parsing time! — zmo, Mar 09 '16 at 19:47
here you go, as promised I gave a few examples. Though as I say in my update, the idea to use a custom constructor to convert all `dict`s into `namedtuple`s is not as much a good idea as I thought when I wrote the comment. — zmo, Mar 11 '16 at 23:29

Anthon · Answer 2 · 2016-03-09T18:50:22.613

At the cost of not as easily being able to iterate over the different mapping keys after assigning them as attributes you can do the following:

from __future__ import print_function

import errno
import sys
import yaml

backup_conf="""
loglevel: INFO
username: root
password: globalsecret
destdir: /dsk/bckdir/
avoidprojects:

matchregex: /bkp/

depots:
    server1:
        password: asecret

    server2:
        username: root

    server3:

    server4:
        destdir: /disk2/bkp/

projects:
    proj1:
        matchregex:
            - /backups/
            - /bkp/
"""

class Struct:
    pass

    def __repr__(self):
        res = {}
        for x in dir(self):
            if x.startswith('__'):
                continue
            res[x] = getattr(self, x)
        return repr(res)


def assign_dict_as_attr(obj, d):
    assert isinstance(d, dict)
    for key in d:
        value = d[key]
        if isinstance(value, dict):
            x = Struct()
            setattr(obj, key, x)
            assign_dict_as_attr(x, value)
        else:
            setattr(obj, key, value)

class Config:

    def __init__(self, filename="backup.cfg", data=None):
        self.cfg = {}
        if data is None:
            try:
                fd = open(filename,'r')
                try:
                    yamlcfg = yaml.safe_load(fd)
                except yaml.YAMLError as e:
                    sys.exit(e.errno)
                finally:
                    fd.close()
            except ( IOError, OSError ) as e:
                sys.exit(e.errno)
        else:
            try:
                yamlcfg = yaml.safe_load(data)
            except yaml.YAMLError as e:
                sys.exit(e.errno)

        print('yamlcfg', yamlcfg)
        assign_dict_as_attr(self, yamlcfg)


if __name__ == '__main__':
    # Read config file to cfg
    config = Config(data=backup_conf)
    print('loglevel', config.loglevel)
    print('depots.server1', config.depots.server1)
    print('depots.server1.password', config.depots.server1.password)

to get:

loglevel INFO
depots.server1 {'password': 'asecret'}
depots.server1.password asecret

Another solutions is to make the __getattr__() somewhat smarter:

class Struct:
    def __init__(self, d):
        self._cfg = d

    def __getattr__(self, name):
        res = self._cfg[name]
        if isinstance(res, dict):
            res = Struct(res)
        return res

    def __str__(self):
        res = {}
        for x in self._cfg:
            if x.startswith('__'):
                continue
            res[x] = self._cfg[x]
        return repr(res)


class Config:

    def __init__(self, filename="backup.cfg", data=None):
        self.cfg = {}
        if data is None:
            try:
                fd = open(filename,'r')
                try:
                    self._cfg = yaml.safe_load(fd)
                except yaml.YAMLError as e:
                    sys.exit(e.errno)
                finally:
                    fd.close()
            except ( IOError, OSError ) as e:
                sys.exit(e.errno)
        else:
            try:
                self._cfg = yaml.safe_load(data)
            except yaml.YAMLError as e:
                sys.exit(e.errno)


    def __getattr__(self, name):
        res = self._cfg[name]
        if isinstance(res, dict):
            res = Struct(res)
        return res



if __name__ == '__main__':
    # Read config file to cfg
    config = Config(data=backup_conf)
    print('loglevel', config.loglevel)
    print('depots.server1', config.depots.server1)
    print('depots.server1.password', config.depots.server1.password)

Which gives you the same output as before.

score 0 · Answer 3 · answered Mar 10 '16 at 12:32

0

Just use easydict in combination with anyconfig.

answered Mar 10 '16 at 12:32

user1340544

21
1

Trying to make a dict behave like a clean class/method structure

3 Answers3

① Using constructors

② Using namedtuples

③ Using a custom yaml.Loader to build namedtuples

Conclusion

② Using `namedtuple`s

③ Using a custom `yaml.Loader` to build `namedtuple`s