updating global variable between two threads

Question

I need to extract information using tweepy on two steps and update global variable location at the first iteration (check PART1 in my code); then do some data manipulation to return list of locations (or update it), then apply the second part where I extract data from twitter (PART2).

Here is my code:

locations=[[-6.38,49.87,1.77,55.81], [-3.38,39.87,1.77,55.81]]

def Part1():
# Doing something here to get locations, then update the location variable 

def Part2():
    for l in locations:
        # then I need to delete the location (l) from the gloabl list 

t1 = Thread(target = Part1)
t2 = Thread(target = Part2)

def main():
    t1.start()
    t2.start()

Is this the best way to do it ?. What is the recommended way to make location as a gloabl variable and update/consume it in the both threads.

In Python it is said that you should never modify a list while iterating over it. I guess if you are iterating over a list in one thread and other thread remove an element you may have the same kind of problem. — Paulo Scardine, Jan 19 '15 at 20:16
What about not using threads? Just call Part1(), update locations and then Part2() — BurningKarl, Jan 19 '15 at 20:32

score 1 · Answer 1 · edited May 23 '17 at 12:06

There are two issues here, first you should use a Queue for locations instead of a list. This question can help explain why Queues should be preferred in a threaded environment.

Second, as Paulo mentioned in the comments, using a for ... in ... construct on a list while it is being modified is a bad idea. The iterator created will be unaware of changes to the list, so it might return the same element many times or skip items depending on how you modify the list. So instead of using an iterator over the list, we use a while loop and pop items.

Your example might look something like this

import threading
import queue # Queue in python2.x

def Part1(locations):
    # Do work to get locations
    locations.put(location)

def Part2(locations):
    # setup API

    while True:
        try:
            # Get a location and do something with it
            location = locations.get()
            ...

        except queue.Empty:
            # No items in locations, quit
            break

def main():
    # setup queue and initial values
    locations = queue.Queue()
    locations.put([-6.38,49.87,1.77,55.81])

    # start the threads
    t1 = threading.Thread(target=Part1, args=(locations,))
    t2 = threading.Thread(target=Part2, args=(locations,))
    t1.start()
    t2.start()

Note that as written, this example assumes Part1 generates locations faster than Part2 consumes them. If locations ever becomes empty, Part2 will quit even if Part1 later adds more objects. If that's not the case you will have to change the break statement to a sleep and add some way for Part1 to signal to Part2 that it has finished.

updating global variable between two threads

1 Answers1