How to realize a RW lock with multithreading gRPC?

Question

I am trying to write a simple multi-threading server-client stock trading Python program with gRPC and concurrent.future.ThreadPoolExecutor

The server will run a specific number of threads, performs Lookup() and Trade() request sent from clients. The server will maintain a list. That means Lookup() should set read lock, and Trade() should set write lock.

However, it seems that the documentation of gRPC doesn't mention anything about RW lock. Is ThreadPoolExecutor thred-safe?

Any suggestion is appreciated!

gRPC is a protocol for sending stuff over the network. What resource are you locking? — Schwern, Feb 20 '23 at 04:27
Hi. The server will keep reading or writing to a list/dict, so I need to lock it when doing reading/writing @Schwern — mino, Feb 20 '23 at 04:30
Hi. gRPC doesn't store your list/dict. It talks to the thing that does. This sounds like an issue for your storage service, most support some sort of locking. Where are you storing your list/dict? — Schwern, Feb 20 '23 at 04:49
@Schwern It is just a simple list/dict store in the memory. Well I think I need to elaborate my question. So I know gRPC is just a protocal for the network, this network will take multiple threads, created by `ThreadPoolExecutor`. So now my question is, when multiple threads need to read/write this global dict/list, some sort of lock is required. But I don't see any documentation of ThreadPoolExecutor mentions it. Does that mean `ThreadPoolExecutor` is thread-safe for I/O? — mino, Feb 20 '23 at 05:04
That sounds like a much better way to phrase the question. Consider [editing your question](https://stackoverflow.com/posts/75505035/edit). Also if you have some example code, that would help people understand. Finally, [does this answer help?](https://stackoverflow.com/a/55383158/14660). Note the part about Python's [GIL](https://docs.python.org/3/glossary.html#term-global-interpreter-lock). This is probably what you want: you *do* want to lock (via the GIL) while manipulating data, and you *don't* want to wait for gRPC networking. — Schwern, Feb 20 '23 at 19:22

score 0 · Accepted Answer · answered Feb 20 '23 at 22:59

A threadPoolExecutor with a max_workers greater than 1 requires that your only submit functions that reference thread-safe shared state only. For instance, you should not mutate a global variable from multiple jobs submitted to the thread pool, thus you must protect this with some kind of synchronization primitive.

# /!\ Not thread-safe

from concurrent.futures import ThreadPoolExecutor


def mutate_value():
  global some_value
  some_value += 1


some_value = 0

with ThreadPoolExecutor(max_workers=2) as e:
  e.submit(mutate_value)
  e.submit(mutate_value)

# Thread-safe

from concurrent.futures import ThreadPoolExecutor
from threading import Lock


lock = Lock()


def mutate_value():
  global some_value

  with lock:
    some_value += 1


some_value = 0

with ThreadPoolExecutor(max_workers=2) as e:
  e.submit(mutate_value)
  e.submit(mutate_value)

Thanks! I was looking at GIL. Some people say GIL prevent multhread to be real multithread on Python. But now I see, thx! — mino, Feb 20 '23 at 23:45

How to realize a RW lock with multithreading gRPC?

1 Answers1