Yield & closing file objects properly

Question

I have the following class:

class JsonIterator(object):
    def __init__(self,folder):
        self.root = os.path.join(os.getcwd(), folder)
        self.fis = [fi for fi in os.listdir(folder) if "lyrics" in fi]

    def __iter__(self):
        i = 0
        with open(os.path.join(self.root,self.fis[i])) as f:
            yield json.load(f)
        i += 1

It's not working how I want it to--it seems not to get beyond the first file. I've tried

def __iter__(self):
    i = 0
    f = open(os.path.join(self.root, self.fis[i]))
    js = json.load(f)
    f.close()
    yield js
    i += 1

But either way, len(list(JsonIterator("path/to/myfolder"))) gives me 1, and I know for a fact that there is more than one file in the folder. The alternative, of course, would be something like

def __iter__(self):
    i = 0
    yield json.load(open(os.path.join(self.root, self.fis[i])))
    i += 1

but all those dangling open files suck up so much memory my process gets killed.

So what should I do? I thought about writing some kind of decorator, but I don't really understand how those work, or even if it'd solve my problem.

Thanks!

unutbu · Accepted Answer · 2013-03-26T19:42:00.983

3

You need to loop over the filenames in self.fis:

class JsonIterator(object):
    def __init__(self,folder):
        self.root = os.path.join(os.getcwd(), folder)
        self.fis = [fi for fi in os.listdir(folder) if "lyrics" in fi]

    def __iter__(self):
        for fi in self.fis:
            with open(os.path.join(self.root, fi)) as f:
                obj = json.load(f)
            yield obj

edited Mar 26 '13 at 19:42

answered Mar 26 '13 at 19:34

unutbu

842,883
184
1,785
1,677

I think I would store the result of `json.load()` in a variable, and yield it from outside the context manager, so that the file is closed before the function pauses; otherwise, this is the best answer. – SingleNegationElimination Mar 26 '13 at 19:38

score 0 · Answer 2 · edited May 23 '17 at 11:59

0

As mentioned in the other answers, you need to iterate over your list of files inside of __iter__(). Here is an equivalent alternative that uses a generator expression to do the same thing:

def __iter__(self):
    return (json.load(open(os.path.join(self.root, fi))) for fi in self.fis)

The files will be closed automatically when they are garbage collected, assuming you are using CPython.

edited May 23 '17 at 11:59

Community

1
1

answered Mar 26 '13 at 19:36

Andrew Clark

202,379
35
273
306

files will be closed automatically when they are garbage collected in all python implementations I'm aware of. Only in CPython, which uses reference counting, will the be collected *immediately*, instead of, say, on program exit, or when the program is experiencing load at a later time. – SingleNegationElimination Mar 26 '13 at 19:46

Yield & closing file objects properly

2 Answers2