Monitoring a threaded Python program with htop

Question

First of all, this is the code I am referring to:

from random import randint
import time
from threading import Thread
import Queue

class TestClass(object):
    def __init__(self, queue):
        self.queue = queue

    def do(self):
        while True:
            wait = randint(1, 10)
            time.sleep(1.0/wait)
            print '[>] Enqueuing from TestClass.do...', wait
            self.queue.put(wait)


class Handler(Thread):
    def __init__(self, queue):
        Thread.__init__(self)
        self.queue = queue

    def run(self):
        task_no = 0
        while True:
            task = self.queue.get()
            task_no += 1
            print ('[<] Dequeuing from Handler.run...', task,
                                            'task_no=', task_no)
            time.sleep(1) # emulate processing time
            print ('[*] Task %d done!') % task_no
            self.queue.task_done()

def main():
    q = Queue.Queue()
    watchdog = TestClass(q)
    observer = Thread(target=watchdog.do)
    observer.setDaemon(True)

    handler = Handler(q)
    handler.setDaemon(True)

    handler.start()
    observer.start()

    try:
        while True:
            wait = randint(1, 10)
            time.sleep(1.0/wait)
            print '[>] Enqueuing from main...', wait
            q.put(wait)
    except KeyboardInterrupt:
        print '[*] Exiting...', True

if __name__ == '__main__':
    main()

While the code is not very important to my question, it is a simple script that spawns 2 threads, on top of the main one. Two of them enqueue "tasks", and one dequeues them and "executes" them.

I am just starting to study threading in python, and I have of course ran into the subject of GIL, so I expected to have one process. But the thing is, when I monitor this particular script with htop, I notice not 1, but 3 processes being spawned.

How is this possible?

Answer 1

The GIL means only one thread will "do work" at a time but it doesn't mean that Python won't spawn the threads. In your case, you asked Python to spawn two threads so it did (giving you a total of three threads). FYI, top lists both processes and threads in case this was causing your confusion.

Python threads are useful for when you want concurrency but don't need parallelism. Concurrency is a tool for making programs simpler and more modular; it allows you to spawn a thread per task instead of having to write one big (often messy) while loop and/or use a bunch of callbacks (like JavaScript).

If you're interested in this subject, I recommend googling "concurrency versus parallelism". The concept is not language specific.

Edit: Alternativly, you can just read this Stack Overflow thread.