2

The related question How do I merge two python iterators? works well for two independent iterators. However, I haven't been able to find or think of the tools necessary for merging two iterators where one is recursive and takes the other as an input. I have iterator stuff that is a simple list. Then I have iterator theta that takes a function func and yields x, func(x), func(func(x)), where one of the inputs to func is an element of stuff. I've solved this with mutable state as follows:

theta = some_initial_theta
for thing in stuff:
    theta = update_theta(theta, thing)
return theta

A concrete example in this format:

def update_theta(theta, thing):
    return thing * 2 + theta

stuff = [100, 200, 300, 400]


def my_iteration():
    theta = 0
    for thing in stuff:
        theta = update_theta(theta, thing)
    print(theta)
# This prints 2000

I'm sure there's an elegant way of doing this without the mutable state and the for loop. A simple zip doesn't do it for me because the theta iterator uses its previous element as an input to the next element.

One elegant way of expressing theta is using the iterate method available in the more_itertools package:

iterate(lambda theta: update_theta(theta, thing), some_initial_theta)

However, the problem with this is that thing will be fixed throughout the iteration. It would be possible to deal with this by passing in the entire list stuff and then return the remainder of it from the update_theta method:

iterate(lambda theta: update_theta(theta[0], theta[1]), (some_initial_theta, stuff))

However, I'd really rather not modify the update_theta method to take an entire list it's not interested in and deal with the mechanics of returning the tail of that list. While it's programmatically not difficult, it's poor separation of concerns. update_theta shouldn't know anything about or care about the entire list stuff.

Community
  • 1
  • 1
Xander Dunn
  • 2,349
  • 2
  • 20
  • 32
  • 2
    Can you provide a simple self-contained example showing what you're talking about (e.g., actual examples of simple iterators that behave in the way you describe)? – BrenBarn Nov 08 '15 at 23:36
  • @BrenBarn I added a concrete example. – Xander Dunn Nov 08 '15 at 23:51
  • 1
    See [**`reduce`**](https://docs.python.org/2/library/functions.html#reduce) – Peter Wood Nov 08 '15 at 23:52
  • @PeterWood how will you deal with taking a new element of the list `stuff` on each step of the reduce? – Xander Dunn Nov 08 '15 at 23:54
  • I don't really see anything wrong with the way you're doing it. Also from your example it seems you're trying to combine the two iterators to get a single result (not an iterable of intermediate results), which is somewhat different from the question you linked to, and means that the details of what goes on along the way to that result are probably less important. – BrenBarn Nov 09 '15 at 00:04
  • @thoughtadvances: `reduce` just does that. It's exactly what `reduce` was designed for. – user2357112 Nov 09 '15 at 00:16

1 Answers1

4

As Peter Wood suggests in the comments, this is exactly what the built-in function reduce does:

result = reduce(update_theta, stuff, some_initial_theta)

In Python 3, reduce has been moved to functools.reduce, so you'd need to import that:

from functools import reduce

If you want an iterator of all the intermediate values, Python 3 provides itertools.accumulate. There's no argument to specify an initial value, so you'd need to put the initial value in the iterator:

from itertools import accumulate, chain
result_iterator = accumulate(chain([some_initial_theta], stuff), update_theta)

Python 2 doesn't have itertools.accumulate, but you could copy the equivalent code from the Python 3 documentation. There's no easy way to formulate it in terms of the Python 2 standard tools, which is why people wanted it added to Python 3 in the first place.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Thanks! I need to test out `reduce` some more. I think what tripped me up with `reduce` earlier is that it handles only 2 parameters. Like x and y in x + y, whereas my update_theta() takes many parameter. I think what I need to do is define a lambda that keeps constant all the other parameters and then simply use it with reduce like a single parameter function. – Xander Dunn Nov 09 '15 at 18:21
  • I've been slow getting back to this, but I believe your solution is the right one, so I'll go ahead and give you credit. I just need to use a lambda to reduce it to two parameters. – Xander Dunn Nov 14 '15 at 00:17