Understanding Python contextvars

Question

Regarding the following SO answer . I've made some changes in order to understand the difference between do use Contextvars and don't.

I expect at some point the variable myid gets corrupted but changing the range to a higher number seems doesn't affect at all.

import asyncio
import contextvars

# declare context var
request_id = contextvars.ContextVar('Id of request.')


async def some_inner_coroutine(myid):
    # get value
    print('Processed inner coroutine of myid   : {}'.format(myid))
    print('Processed inner coroutine of request: {}'.format(request_id.get()))
    if myid != request_id.get():
        print("ERROR")


async def some_outer_coroutine(req_id):
    # set value
    request_id.set(req_id)

    await some_inner_coroutine(req_id)

    # get value
    print('Processed outer coroutine of request: {}'.format(request_id.get()))


async def main():
    tasks = []
    for req_id in range(1, 1250):
        tasks.append(asyncio.create_task(some_outer_coroutine(req_id)))

    await asyncio.gather(*tasks)


if __name__ == '__main__':
    asyncio.run(main())

Results

Processed inner coroutine of myid   : 1
Processed inner coroutine of request: 1
Processed outer coroutine of request: 1
Processed inner coroutine of myid   : 2
Processed inner coroutine of request: 2
Processed outer coroutine of request: 2
Processed inner coroutine of myid   : 3
Processed inner coroutine of request: 3
Processed outer coroutine of request: 3
Processed inner coroutine of myid   : 4
Processed inner coroutine of request: 4
Processed outer coroutine of request: 4
...
...
Processed inner coroutine of myid   : 1244
Processed inner coroutine of request: 1244
Processed outer coroutine of request: 1244
Processed inner coroutine of myid   : 1245
Processed inner coroutine of request: 1245
Processed outer coroutine of request: 1245
Processed inner coroutine of myid   : 1246
Processed inner coroutine of request: 1246
Processed outer coroutine of request: 1246
Processed inner coroutine of myid   : 1247
Processed inner coroutine of request: 1247
Processed outer coroutine of request: 1247
Processed inner coroutine of myid   : 1248
Processed inner coroutine of request: 1248
Processed outer coroutine of request: 1248
Processed inner coroutine of myid   : 1249
Processed inner coroutine of request: 1249
Processed outer coroutine of request: 1249

What should I change to see an unexpected behaviour of the variable myid?

Why create them as tasks, which start running immediately, if you then gather them? Just do the latter — Pynchia, Jul 26 '20 at 21:48
contextvars are not a replacement for local variables, those work just fine as they are. They are a replacement for **global** variables, which are shared among all tasks. If `myid` were a global variable (say set through a context manager), it would be able to hold only a single value. With threads you can introduce a thread-local variable and have a "global" state that is local to the thread. Contextvars are the equivalent for tasks (and also support callbacks). — user4815162342, Jul 27 '20 at 06:55
So basically my tasks are running in the same thread and sharing the same context var. And if I would had more than one thread, then I see how useful is context vars. Am I right? — Ricardo, Jul 27 '20 at 07:29
No. The point of a contextvar is for a single var to have different values in different asyncio tasks and task-like contexts, all running on the same thread. contextvars are **not** useful for multithreading because they are designed for asyncio which is explicitly single-threaded. — user4815162342, Jul 27 '20 at 09:36
Thanks for the explanation, what should I change of my code to see how useful is contextvars? — Ricardo, Jul 27 '20 at 15:46
It's not clear exactly what you're trying to demonstrate. You could use a global variable and would quickly notice that the same variable is observed (and overwritten by) all tasks. — user4815162342, Jul 28 '20 at 11:51

alex_noname · Accepted Answer · 2021-08-07T16:03:53.083

Context variables are convenient when you need to pass a variable along the chain of calls so that they share the same context, in the case when this cannot be done through a global variable in case of concurrency. Context variables can be used as an alternative to global variables both in multi-threaded code and in asynchronous (with coroutines).

Context variables are natively supported in asyncio and are ready to be used without any extra configuration. Because when a Task is created it copies the current context and later runs its coroutine in the copied context:

# asyncio/task.py
class Task:
    def __init__(self, coro):
        ...
        # Get the current context snapshot.
        self._context = contextvars.copy_context()
        self._loop.call_soon(self._step, context=self._context)

    def _step(self, exc=None):
        ...
        # Every advance of the wrapped coroutine is done in
        # the task's context.
        self._loop.call_soon(self._step, context=self._context)
        ...

Below is your example showing the corruption of a global variable compared to context vars:

import asyncio
import contextvars

# declare context var
current_request_id_ctx = contextvars.ContextVar('')
current_request_id_global = ''


async def some_inner_coroutine():
    global current_request_id_global

    # simulate some async work
    await asyncio.sleep(0.1)

    # get value
    print('Processed inner coroutine of request: {}'.format(current_request_id_ctx.get()))
    if current_request_id_global != current_request_id_ctx.get():
        print(f"ERROR! global var={current_request_id_global}")


async def some_outer_coroutine(req_id):
    global current_request_id_global

    # set value
    current_request_id_ctx.set(req_id)
    current_request_id_global = req_id

    await some_inner_coroutine()

    # get value
    print('Processed outer coroutine of request: {}\n'.format(current_request_id_ctx.get()))


async def main():
    tasks = []
    for req_id in range(1, 10000):
        tasks.append(asyncio.create_task(some_outer_coroutine(req_id)))

    await asyncio.gather(*tasks)


if __name__ == '__main__':
    asyncio.run(main())

Output:

...
Processed inner coroutine of request: 458
ERROR! global var=9999
Processed outer coroutine of request: 458

Processed inner coroutine of request: 459
ERROR! global var=9999
Processed outer coroutine of request: 459
...

An example of converting code that uses threading.local() can be found in PЕP 567

Hmm, this feels very much like "dynamic scoping" which is a "feature" of emacs-lisp that in practice is pretty handy (though in emacs lisp every global is effectively a context var). It's intersting how this "broken" form of scoping - implemented for simplicity is showing up in elsewhere. This concept also shows up in haskell through the context monad I believe. — Att Righ, Aug 30 '22 at 13:43
So my key takeaway here is: `contextvars` only "works" with tasks/event loops which *explicitly and directly* integrate with the machinery of `contextvars` under the hood, deep inside the internals. Correct? A library which ignores `contextvars` in its internals would break usercode which relied upon `contextvars`? — Dubslow, Apr 17 '23 at 17:57
No, you can use `contextvars` without any frameworks like `asyncio`. This module is pretty self-sufficient and provides everything you need to manage contexts. — alex_noname, Apr 17 '23 at 19:27

Understanding Python contextvars

1 Answers1

Linked