Subclassing Future Class

Question

I would like to subclass the Future class of the concurrent Python module.

The docs:

The Future class encapsulates the asynchronous execution of a callable. Future instances are created by Executor.submit().

The docs of Executor don't explain where it takes the Future class from.

... How can make Executor.submit() force to take my custom Future class?

Why do I need it?

I like OOP since it creates readable code. I would like the result to look like this:

for my_future in concurrent.futures.as_completed(...):
    my_future.my_custom_method()

Because by subclassing you potentially mutate the implementation you know nothing about. You might accidentally override methods you know nothing about, and if you don't, it might break on next release. — Elazar, Jul 18 '16 at 22:43
In general, inheritance is last resort. You should favor composition and wrapping/adapting. Even if your reasons are better than syntactic convenience - of a form that is unconventional and therefore only seems to be readable. — Elazar, Jul 18 '16 at 22:45
And you should make sure that the class was designed and documented for subclassing, with guarantees about future changes and overridable methods — Elazar, Jul 18 '16 at 22:48
Of course, for toy projects and experiments do whatever you want. — Elazar, Jul 18 '16 at 22:49
@Elazar I still don't know why you think this is awful. I am curious, please explain what is awful here. Or do you think OOP is always awful? — guettli, Jul 19 '16 at 08:25
This is simply not OOP. It's merely method-call syntax that you like. In order to be able to use this syntax, you use a heavy semantics-oriented mechanism with many consequences, as explained above - inheritance. In other languages you have extension-methods for that; in Python, you can probably assign to `__dict__` — Elazar, Jul 19 '16 at 09:26
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/117687/discussion-between-guettli-and-elazar). — guettli, Jul 19 '16 at 10:49

duyue · Accepted Answer · 2016-07-28T02:48:22.393

Looking at the code, ProcessPoolExecutor.submit() and ThreadPollExecutor.submit(), Excutor.submit() returns an instance of Future, which is defined in conccurent.futures._base.Future.

So, here comes the trick. You can subclass and replace the original Future, and then add custom methods in the subclass.

It's doable, but not recommended. It's better to use composition rather than inheritance for this purpose. There is a good chapter on inheritance and composition in Learn Python the Hard Way

Back to the the question, here is a composition example:

class Myclass(object):
    def __init__(self, workers=3):
         self.executor = concurrent.futures.ProcessPoolExcutor(workers)

    def run(self, job):
        '''Job to be run in a Executor'''

    def submit(self, jobs):
        self.futures = [executor.submit(self, foo, job) for job in jobs]

    def done(self, result):
        '''Dealing with the result'''

    def harvest(self):
        for my_future in concurrent.futures.as_completed(self.futures):
            self.done(my_future.result())

And then you can subclass MyClass and implement different done method.

Yes, this works. I just ask myself why the library was not implemented with subclassing in mind. The method `as_completed()` returns futures. It needs useless code to wrap this. Yes, for this case your sentence "It's better to use composition rather than inheritance for this purpose" applies. But it only applies since the upstream code does not provide an easy way for subclassing. Don't get me wrong. It's not your fault. Thank your for your answer! — guettli, Jul 27 '16 at 07:54
@guettli You're welcome.From https://www.python.org/dev/peps/pep-3148/#rationale, we can know that Python's futures module was heavily influenced by java.util.concurrent package. Don't why upstream doesn't provide an easy way for subclassing, too. Maybe upstream want it simple, maybe subclass is not a major requirement, or maybe it's already done. There is no perfect thing [shrug]. — duyue, Jul 28 '16 at 03:02

score 3 · Answer 2 · edited May 23 '17 at 12:08

The use of the Future concrete class is hard-wired into Executor.submit() (whether for processes or threads). Therefore, I do not think it is possible to do exactly what you are asking. However, you can return any result from the callable passed to Executor.submit(). Therefore, put your custom methods in a custom return class:

class my_result(object):
    def my_custom_method(self):
        pass

def x():
    return my_result()

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(x), executor.submit(x)]
    for my_future in concurrent.futures.as_completed(futures):
        my_future.result().my_custom_method()
               # ^^^^^^^^^

Edit Or, if you really want your inner loop to be clean, change the last two lines to:

    for my_result in (f.result() for f in concurrent.futures.as_completed(futures)):
                   # ^^^^^^^^^^^^^^^^^^^^
        my_result.my_custom_method()

The generator expression (f.result() ... (futures)) takes the iterator of futures from as_completed and gives you an iterator of the results of those futures. You can then loop through those results.

Subclassing Future Class

Why do I need it?

2 Answers2