I'm trying to write a class in Python that behaves as a generator object, particularly in that when it's garbage collected .close() is called on it. That's important because it means that when the generator is interrupted I can make sure it'll clean up after itself, for example closing files or releasing locks.
Here's some explanatory code:
If you interupt a generator, then when it's garbage collected, Python calls .close() on the generator object, which throws a GeneratorExit error into the generator, which can be caught to allow cleanup, like follows:
from threading import Lock
lock = Lock()
def CustomGenerator(n, lock):
lock.acquire()
print("Generator Started: I grabbed a lock")
try:
for i in range(n):
yield i
except GeneratorExit:
lock.release()
print("Generator exited early: I let go of the lock")
raise
print("Generator finished successfully: I let go of the lock")
for i in CustomGenerator(100, lock):
print("Received ", i)
time.sleep(0.02)
if i==3:
break
if not lock.acquire(blocking=False):
print("Oops: Finished, but lock wasn't released")
else:
print("Finished: Lock was free")
lock.release()
Generator Started: I grabbed a lock
Received 0
Received 1
Received 2
Received 3
Generator exited early: I let go of the lock
Finished: Lock was free
However, if you try to implement your own generator object by inheriting from collections.abc.Generator, Python doesn't seem to notice that it should call close when the object is collected:
from collections.abc import Generator
class CustomGeneratorClass(Generator):
def __init__(self, n, lock):
super().__init__()
self.lock = lock
self.lock.acquire()
print("Generator Class Initialised: I grabbed a lock")
self.n = n
self.c = 0
def send(self, arg):
value = self.c
if value >= self.n:
raise StopIteration
self.c += 1
return value
def throw(self, type, value=None, traceback=None):
print("Exception Thrown in Generator: I let go of the lock")
self.lock.release()
raise StopIteration
for i in CustomGeneratorClass(100, lock):
print("Received ", i)
time.sleep(0.02)
if i==3:
break
if not lock.acquire(blocking=False):
print("Oops: Finished, but lock wasn't released")
else:
print("Finished: Lock was free")
lock.release()
Generator Class Initialised: I grabbed a lock
Received 0
Received 1
Received 2
Received 3
Oops: Finished, but lock wasn't released
I thought that inheriting Generator would be sufficient to convince python that my CustomGeneratorClass was a generator and should have .close() called on it when garbage collected.
I assume this has something to do with the fact that while 'generator object' are some kind of special Generator:
from types import GeneratorType
c_gen = CustomGenerator(100)
c_gen_class = CustomGeneratorClass(100)
print("CustomGenerator is a Generator:", isinstance(c_gen, Generator))
print("CustomGenerator is a GeneratorType:",isinstance(c_gen, GeneratorType))
print("CustomGeneratorClass is a Generator:",isinstance(c_gen_class, Generator))
print("CustomGeneratorClass is a GeneratorType:",isinstance(c_gen_class, GeneratorType))
CustomGenerator is a Generator: True
CustomGenerator is a GeneratorType: True
CustomGeneratorClass is a Generator: True
CustomGeneratorClass is a GeneratorType: False
Can I make a user defined class object that is GeneratorType?
Is there something I don't understand about how python decides what to call .close() on?
How can I ensure that .close() is called on my custom generator?
This question is not a duplicate of How to write a generator class. For actually making a generator class, the accepted answer for that question does recommends exactly the structure I'm trying here, which is a generator class but is not correctly garbage collected, as shown in the code above.
