Avoiding for loops using list comprehension in Python

Question

Suppose I have this pandas.Series:

import pandas as pd
returns = pd.Series([1,2,3,4,5])

And I want to do some simple mathematics stuff on it and save the results in a new list. So I 'll explain them. For the first item I want to calculate Variance of returns items:

import numpy as np
def First(Ser):
    return np.var(Ser)

For rest of them I want to do this:

def rest(variances , returns , i):
    return 0.94 * variances[i-1] + ( 1 - 0.94 ) * (returns[i-1])**2

Then using list comprehension:

variances = [rest(variances , returns , i) for i in range(len(returns)) if i!=0 else First(returns)]

But it gives me this error:

variances = [rest(variances , returns , i) for i in range(len(returns)) if i!=0 else First(returns)]
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ^
SyntaxError: invalid syntax

How can I avoid using a for loop for this purpose?
Full script:

import pandas as pd
import numpy as np

returns = pd.Series([1,2,3,4,5])

def First(Ser):
    return np.var(Ser)
    
def rest(variances , returns , i):
    return 0.94 * variances[i-1] + ( 1 - 0.94 ) * (returns[i-1])**2
    
variances = [rest(variances , returns , i) for i in range(len(returns)) if i!=0 else First(returns)]

More explaination:

What is variances?
variances is a list that I want to be built with the List Comprehension technique! I want to use variances simultaneously as it is created. My basic goal is to avoid using a for loop.

Algorithm & Expected output with use of for loop:

import pandas as pd
import numpy as np

returns = pd.Series([1,2,3,4,5])
variances = []

for item in range(len(returns)):
    if item == 0:
        variances.append(np.var(returns))
    else:
        variances.append(0.94*variances[item-1] + (1-0.94)*(returns[item-1])**2)

print(variances)

[2.0, 1.94, 2.0636, 2.4797840000000004, 3.290996960000001]

Thank you for posting a good question. It includes a proper problem description, a question, your own code in form of a minimal reproducible example and a description of what errors occur. I wish more people would follow the same pattern. — Finomnis, Aug 29 '21 at 09:44
@U12-Forward its a list that is creating with list comprehension فechnique. `variances = [rest(variances , returns , i) for i in range(len(returns)) if i!=0 else First(returns)]` — Shayan, Aug 29 '21 at 09:47
Does this answer your question? [if/else in a list comprehension](https://stackoverflow.com/questions/4260280/if-else-in-a-list-comprehension) — Finomnis, Aug 29 '21 at 09:48
@Finomnis thanks. it would be great to vote it up if you found it useful. — Shayan, Aug 29 '21 at 09:49
@Shayan why not? It answers the question why you get the `invalid syntax` error — Finomnis, Aug 29 '21 at 09:50
@Shayan I will vote it up if you add the definition of `variances` ;) — Finomnis, Aug 29 '21 at 09:51
@Finomnis if you check my code, my statement after `else` isn't string! it should be another function! — Shayan, Aug 29 '21 at 09:53
@Shayan have you even read the responses to the linked question? The order of your statements in the list comprehension is wrong, it should be `[f(x) if condition else g(x) for x in sequence]`. I fail to see how the type of the statement would make any difference. — Finomnis, Aug 29 '21 at 09:56
@Shayan sergiomahi even answered that below. Please don't dismiss comments prematurely. — Finomnis, Aug 29 '21 at 09:58
You have a fundamental misunderstanding of how list comprehensions work. We could help you if you would post what the expected outcome of your code is supposed to be. Like, if your code would run successfully, what values are supposed to be in `variances` at the end? That would help us in identifying what you are trying to do. — Finomnis, Aug 29 '21 at 10:07
With your `for` loop example and the expected outcome, I now understand what the goal of your algorithm was. Sadly, I think using the unrolled `for` loop is the correct approach. As the iteration references other items in the output list, it is unsuitable for a list comprehension. If I would have to implement your problem, I would also have used a normal `for` loop. — Finomnis, Aug 29 '21 at 10:24
@Finomnis just I want to thank you! it's so clear! I don't understand why should you be pessimistic about someone is trying to say thanks for your attention to my problem! — Shayan, Aug 29 '21 at 10:25
As a further explanation: List comprehensions are meant for mapping problems where every item can be computed independently from all other items. This is sadly not the case for your problem, and that is why I think list comprehension is the incorrect tool here. — Finomnis, Aug 29 '21 at 10:28
There could be, but I still think the for loop is the most elegant solution. Why do you want to get rid of it? Like, what's your motivation? — Finomnis, Aug 29 '21 at 10:40
it's utilizing higher speed! I want to practice **Functional Programing** (that you know isn't easy). I decided to begin with list comprehensions. after that for more speed, using functional programing. — Shayan, Aug 29 '21 at 11:09

score 3 · Accepted Answer · answered Aug 29 '21 at 10:41

3

You are referring to the i-1'th element of a list (variances) you are creating via list comprehension which I think is not possible. However you could try something like this:

import pandas as pd
import numpy as np


def rest(variances , returns , i):
    if i == 0:
        variances[i] = np.var(returns)
        return variances [i]
    else:
        variances[i] = 0.94 * variances[i-1] + ( 1 - 0.94 ) * (returns[i-1])**2
        return variances [i]

returns = pd.Series([1,2,3,4,5])
placeholder = np.zeros(len(returns))

variances = [rest(placeholder , returns , i) for i in range(len(returns))]

print(variances)

# [2.0, 1.94, 2.0636, 2.4797840000000004, 3.290996960000001]

print(placeholder)

# [2.         1.94       2.0636     2.479784   3.29099696]

You do not necessarily need to return the values in the rest function, as the actual purpose of the function is to update the existing placeholder list. However, if your function does not return anything, the list comprehension will return a list of None values.

answered Aug 29 '21 at 10:41

tjruesch

58
6

Agree, this is also what came to my mind, if you forcefully want to avoid the for loop. The question is, though, if that is more elegant or easiert to read/maintain than the for loop. – Finomnis Aug 29 '21 at 10:48
You technically don't even need the entire placeholder array, you only need a single variable to carry over the [i-1] element, the rest doesn't get reused. – Finomnis Aug 29 '21 at 10:50
It is? Oh that must be because of the `.append`. Would you mind checking how fast the for-loop is with a pre-allocated object and direct [i] writes? – Finomnis Aug 29 '21 at 11:00
Thanks! it's one of the potential solutions! nice one. Thanks a lot ♥ – Shayan Aug 29 '21 at 11:16
I tested this. The list comprehension is not twice as fast. It's barely faster, and in some forms slightly slower. List comprehension is a convenient way of writing an iteration that collects results in a list, but it is not a significant time saver. Wrapping the action in a function, as done here, is certainly better than performing some convoluted if/else inline. It shouldn't be confused with array "vectorization" that moves iterations to compiled code. – hpaulj Aug 29 '21 at 16:03
Thanks for pointing that out, @hpaulj, I did not test the speed properly the first time. Not sure what happened but somehow I messed up the speedtest. I'll delete the comment to avoid confusion. – tjruesch Aug 30 '21 at 08:54

score 1 · Answer 2 · answered Aug 29 '21 at 09:47

1

You are misplacing the if condition:

variances = [rest(variances , returns , i) if i!=0 else First(returns)
 for i in range(len(returns)) ]

answered Aug 29 '21 at 09:47

sergiomahi

964
2
8
21

this gives me : `**NameError**: name 'variances' is not defined`. but I defined variances in `variances = [rest(variances , returns , i) for I in range(len(returns)) if i!=0 else First(returns)]`. I want to use `variances` simultaneously as it is creating! – Shayan Aug 29 '21 at 09:59
You can't use something simultaneous to creating it, that is a misconception. There is a temporal order to things, and if doing an `=` assignment, the right side gets evaluated before it gets written to the left side, so while the list comprehension gets executed, the `variances` variable does not exist yet. What is the expected output of your algorithm? – Finomnis Aug 29 '21 at 10:00
@Finomnis Thanks for your explanations! I updated the question with my expected output! – Shayan Aug 29 '21 at 10:12

score 0 · Answer 3 · answered Aug 29 '21 at 09:48

0

IIUC You could try np.where:

variances = np.where(returns != 0, rest(variances , returns.tolist(), returns.index), First(returns.tolist()))

answered Aug 29 '21 at 09:48

U13-Forward

69,221
14
89
114

Thanks. but unfortunately I didn't got your point exactly. I updated my question and further explanation added as well. – Shayan Aug 29 '21 at 11:07

Avoiding for loops using list comprehension in Python

More explaination:

3 Answers3