Why can't I iterate over an object which delegates via __getattr__ to an iterable?

Question

An example from the book Core Python Programming on the topic Delegation doesn't seem to be working.. Or may be I didn't understand the topic clearly..

Below is the code, in which the class CapOpen wraps a file object and defines a modified behaviour of file when opened in write mode. It should write all strings in UPPERCASE only.

However when I try to open the file for reading, and iterate over it to print each line, I get the following exception:

Traceback (most recent call last):
  File "D:/_Python Practice/Core Python Programming/chapter_13_Classes/
        WrappingFileObject.py", line 29, in <module>
    for each_line in f:
TypeError: 'CapOpen' object is not iterable

This is strange, because although I haven't explicitly defined iterator methods, I'd expect the calls to be delegated via __getattr__ to the underlying file object. Here's the code. Have I missed anything?

class CapOpen(object):
    def __init__(self, filename, mode='r', buf=-1):
        self.file = open(filename, mode, buf)

    def __str__(self):
        return str(self.file)

    def __repr__(self):
        return `self.file`

    def write(self, line):
        self.file.write(line.upper())

    def __getattr__(self, attr):
        return getattr(self.file, attr)


f = CapOpen('wrappingfile.txt', 'w')
f.write('delegation example\n')
f.write('faye is good\n')
f.write('at delegating\n')
f.close()

f = CapOpen('wrappingfile.txt', 'r')

for each_line in f:   # I am getting Exception Here..
    print each_line,

I am using Python 2.7.

Answer 1

This is a non-intuitive consequence of a Python implementation decision for new-style classes:

In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass...

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

This is also explicitly pointed out in the documentation for __getattr__/__getattribute__:

Note This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See Special method lookup for new-style classes.

In other words, you can't rely on __getattr__ to always intercept your method lookups when your attributes are undefined. This is not intuitive, because it is reasonable to expect these implicit lookups to follow the same path as all other clients that access your object. If you call f.__iter__ directly from other code, it will resolve as expected. However, that isn't the case when called directly from the language.

The book you quote is pretty old, so the original example probably used old-style classes. If you remove the inheritance from object, your code will work as intended. That being said, you should avoid writing old style classes, since they will become obsolete in Python 3. If you want to, you can still maintain the delegation style here by implementing __iter__ and immediately delegating to the underlying self.file.__iter__.

Alternatively, inherit from the file object directly and __iter__ will be available by normal lookup, so that will also work.

Answer 2

For an object to be iterable, its class has to have __iter__ or __getitem__ defined.

__getattr__ is only called when something is being retrieved from the instance, but because there are several ways that iteration is supported, Python is looking first to see if the appropriate methods even exist.

Try this:

class Fake(object):
    def __getattr__(self, name):
        print "Nope, no %s here!" % name
        raise AttributeError

f = Fake()
for not_here in f:
    print not_here

As you can see, the same error is raised: TypeError: 'Fake' object is not iterable.

If you then do this:

print '__getattr__' in Fake.__dict__
print '__iter__' in Fake.__dict__
print '__getitem__' in Fake.__dict__

You can see what Python is seeing: that neither __iter__ nor __getitem__ exist, so Python does not know how to iterate over it. While Python could just try and then catch the exception, I suspect the reason why it does not is that catching exceptions is quite a bit slower.

See my answer here for the many ways to make an iterator.