What is the correct way to override the __dir__ method?

Question

This question is meant to be more about __dir__ than about numpy.

I have a subclass of numpy.recarray (in python 2.7, numpy 1.6.2), and I noticed recarray's field names are not listed when diring the object (and therefore ipython's autocomplete doesn't work).

Trying to fix it, I tried overriding __dir__ in my subclass, like this:

def __dir__(self):
    return sorted(set(
               super(MyRecArray, self).__dir__() + \
               self.__dict__.keys() + self.dtype.fields.keys()))

which resulted with: AttributeError: 'super' object has no attribute '__dir__'. (I found here this should actually work in python 3.3...)

As a workaround, I tried:

def __dir__(self):
    return sorted(set(
                dir(type(self)) + \
                self.__dict__.keys() + self.dtype.fields.keys()))

As far as I can tell, this one works, but of course, not as elegantly.

Questions:

1. Is the latter solution correct in my case, i.e. for a subclass of recarray?
2. Is there a way to make it work in the general case? It seems to me it wouldn't work with multiple inheritance (breaking the super-call chain), and of course, for objects with no __dict__...
3. Do you know why recarray does not support listing its field names to begin with? mere oversight?

Answer 1

Python 2.7+, 3.3+ class mixin that simplifies implementation of __dir__ method in subclasses. Hope it will help. Gist.

import six
class DirMixIn:
    """ Mix-in to make implementing __dir__ method in subclasses simpler
    """

    def __dir__(self):
        if six.PY3:
            return super(DirMixIn, self).__dir__()
        else:
            # code is based on
            # http://www.quora.com/How-dir-is-implemented-Is-there-any-PEP-related-to-that
            def get_attrs(obj):
                import types
                if not hasattr(obj, '__dict__'):
                    return []  # slots only
                if not isinstance(obj.__dict__, (dict, types.DictProxyType)):
                    raise TypeError("%s.__dict__ is not a dictionary"
                                    "" % obj.__name__)
                return obj.__dict__.keys()

            def dir2(obj):
                attrs = set()
                if not hasattr(obj, '__bases__'):
                    # obj is an instance
                    if not hasattr(obj, '__class__'):
                        # slots
                        return sorted(get_attrs(obj))
                    klass = obj.__class__
                    attrs.update(get_attrs(klass))
                else:
                    # obj is a class
                    klass = obj

                for cls in klass.__bases__:
                    attrs.update(get_attrs(cls))
                    attrs.update(dir2(cls))
                attrs.update(get_attrs(obj))
                return list(attrs)

            return dir2(self)

Answer 2

Have you tried:

def __dir__(self):
    return sorted(set(
               dir(super(MyRecArray, self)) + \
               self.__dict__.keys() + self.dtype.fields.keys()))

Answer 3

1. and 3: Yes your solution is correct. recarray does not define __dir__ simply because the default implementation was okay, so they didn't bother implementing it, and numpy's devs did not design the class to be subclassed, so I don't see why they should have bothered.

   It's often a bad idea to subclass built-in types or classes that are not specifically designed for inheritance, thus I'd suggest you to use delegation/composition instead of inheritance, except if there is a particular reason(e.g. you want to pass it to a numpy function that excplicitly checks with isinstance).

2. No. As you pointed out in python3 they changed the implementation so that there is an object.__dir__, but on other python versions I can't see anything that you can do. Also, again, using recarray with multiple-inheritance is simply crazy, things will break. Multiple-inheritance should be carefully designed, and usually classes are specifically designed to be used with it(e.g. mix-ins). So I wouldn't bother treating this case, since whoever tries it will be bitten by other problems.

   I don't see why you should care for classes that do not have __dict__... since your subclass has it how should it break? When you'll change the subclass implementation, e.g. using __slots__ you could easily change the __dir__ also. If you want to avoid redefining __dir__ you can simply define a function that checks for __dict__ then for __slots__ etc. Note however that attributes can be generated in subtle ways with __getattr__ and __getattribute__ and thus you simply can't reliably catch all of them.