Complex List Comprehension in Python

Question

I have a list of records in mylist and I want to iterate through the list until I find the first record that satisfies all 3 rules in my rules list. I'm trying to use a nested list comprehension since I thought it would be faster than a nested for loop, however, my logic is a bit complex and I'm not sure if a list comprehension is possible. Here is my code:

import operator
import timeit


def check(rec, vl, op, vr):
    operations = {'>': operator.gt,
                  '<': operator.lt,
                  '>=': operator.ge,
                  '<=': operator.le,
                  '=': operator.eq,
                  '!=': operator.ne}
    return operations[op](vl, vr)


mylist = [{'criteria': {'shipping_point': '3000', 'from_tot_weight': '250', 'to_tot_weight': '999999'}, 'result': {'ship_type': '02'}}, {'criteria': {'shipping_point': '3200', 'from_tot_weight': '350', 'to_tot_weight': '999999'}, 'result': {'ship_type': '02'}}]

rules = [{'varL': ['rec', 'criteria', 'shipping_point'], 'operator': '=', 'varR': "3000"},
         {'varL': ['rec', 'criteria', 'from_tot_weight'], 'operator': '<=', 'varR': "250"},
         {'varL': ['rec', 'criteria', 'to_tot_weight'], 'operator': '>=', 'varR': "250"}]



def run_1():
    newlist = [rec for rec in mylist if all(check(rec, locals()[rule['varL'][0]][rule['varL'][1]][rule['varL'][2]],
                                                   rule['operator'],
                                                   rule['varR'])
                                             for rule in rules)]
    print(newlist)

def run_2():
    found = False
    result = []
    for rec in mylist:
        for rule in rules:
            if check(rec, locals()[rule['varL'][0]][rule['varL'][1]][rule['varL'][2]],
                                   rule['operator'],
                                   rule['varR']):
                found = True
            else:
                found = False
                break
        if found:
            result = rec
            break
    print(result)

run_count = 1
print("==========List Comprehension with Locals()==========")
print(timeit.timeit(run_1, number=run_count))
print(" ")
print("==========Nested for loops==========================")
print(timeit.timeit(run_2, number=run_count))

In this code my list comprehension does not stop when it finds a record that satisfies all the rules in the rules list. Actually, if you change both records in mylist to shipping_point = 3000 then the list comprehension will produce incorrect results. I think that is why it takes longer than the nested for loop. Is it even possible to convert this nested for loop into a nested list comprehension? thank you

*"I'm trying to use a nested list comprehension since I thought it would be faster than a nested for loop"* - List comprehensions aren't magic; they work exactly the same way as loops, it's just a different syntax. The performance difference will be marginal. — kaya3, Mar 04 '20 at 18:44

score 3 · Answer 1 · answered Mar 04 '20 at 18:46

3

List comprehension is only faster when you are actually building a list, and only slightly so (source). In your case, you are only doing a lookup. The nested for loops will not only be faster, but will also make your code more readable.

answered Mar 04 '20 at 18:46

noslenkwah

1,702
1
17
26

Thanks for the reply. My only concern with the nested for loop is that some of the lists may have over 200k records in it. Do you see a problem with that? Does it make sense to create a subset of the list using a list comprehension with some selection criteria and then loop through the filtered list to apply the rules. For example, if there are 200k records in a list and 100k are for shipping_point 3000 and the other 100k are for shipping_point 4000 then I could create a new list for only the 3000's using a list comprehension and then loop through them to apply the rules. Thoughts? – Tony Raimo Mar 04 '20 at 23:23
@TonyRaimo - List comprehension is a for loop anyways. It's just syntactic sugar for **simple** for loops that build lists. There are many issues you can run in to working with a 200K element list, but for-loop vs list-comprehension I don't think will solve any of those. – noslenkwah Mar 04 '20 at 23:36

Complex List Comprehension in Python

1 Answers1