How can I access variables from the caller, even if it isn't an enclosing scope (i.e., implement dynamic scoping)?

Question

Consider this example:

def outer():
    s_outer = "outer\n"

    def inner():
        s_inner = "inner\n"
        do_something()

    inner()

I want the code in do_something to be able to access the variables of the calling functions further up the call stack, in this case s_outer and s_inner. More generally, I want to call it from various other functions, but always execute it in their respective context and access their respective scopes (implement dynamic scoping).

I know that in Python 3.x, the nonlocal keyword allows access to s_outer from within inner. Unfortunately, that only helps with do_something if it's defined within inner. Otherwise, inner isn't a lexically enclosing scope (similarly, neither is outer, unless do_something is defined within outer).

I figured out how to inspect stack frames with the standard library inspect, and made a small accessor that I can call from within do_something() like this:

def reach(name):
    for f in inspect.stack():
        if name in f[0].f_locals:
            return f[0].f_locals[name]
    return None 

and then

def do_something():
    print( reach("s_outer"), reach("s_inner") )

works just fine.

Can reach be implemented more simply? How else can I solve the problem?

Answer 1

There is no and, in my opinion, should be no elegant way of implementing reach since that introduces a new non-standard indirection which is really hard to comprehend, debug, test and maintain. As the Python mantra (try import this) says:

Explicit is better than implicit.

So, just pass the arguments. You-from-the-future will be really grateful to you-from-today.

Answer 2

What I ended up doing was

scope = locals()

and make scope accessible from do_something. That way I don't have to reach, but I can still access the dictionary of local variables of the caller. This is quite similar to building a dictionary myself and passing it on.

Answer 3

We can get naughtier.

This is an answer to the "Is there a more elegant/shortened way to implement the reach() function?" half of the question.

1. We can give better syntax for the user: instead of reach("foo"), outer.foo.

   This is nicer to type, and the language itself immediately tells you if you used a name that can't be a valid variable (attribute names and variable names have the same constraints).

2. We can raise an error, to properly distinguish "this doesn't exist" from "this was set to None".

   If we actually want to smudge those cases together, we can getattr with the default parameter, or try-except AttributeError.

3. We can optimize: no need to pessimistically build a list big enough for all the frames at once.

   In most cases we probably won't need to go all the way to the root of the call stack.

4. Just because we're inappropriately reaching up stack frames, violating one of the most important rules of programming to not have things far away invisibly effecting behavior, doesn't mean we can't be civilized.

   If someone is trying to use this Serious API for Real Work on a Python without stack frame inspection support, we should helpfully let them know.

import inspect


class OuterScopeGetter(object):
    def __getattribute__(self, name):
        frame = inspect.currentframe()
        if frame is None:
            raise RuntimeError('cannot inspect stack frames')
        sentinel = object()
        frame = frame.f_back
        while frame is not None:
            value = frame.f_locals.get(name, sentinel)
            if value is not sentinel:
                return value
            frame = frame.f_back
        raise AttributeError(repr(name) + ' not found in any outer scope')


outer = OuterScopeGetter()

Excellent. Now we can just do:

>>> def f():
...    return outer.x
... 
>>> f()
Traceback (most recent call last):
    ...
AttributeError: 'x' not found in any outer scope
>>> 
>>> x = 1
>>> f()
1
>>> x = 2
>>> f()
2
>>> 
>>> def do_something():
...     print(outer.y)
...     print(outer.z)
... 
>>> def g():
...     y = 3
...     def h():
...         z = 4
...         do_something()
...     h()
... 
>>> g()
3
4

Perversion elegantly achieved.

(P.S. This is a simplified read-only version of a more complete implementation in my dynamicscope library.)

Answer 4

Is there a better way to solve this problem? (Other than wrapping the respective data into dicts and pass these dicts explicitly to do_something())

Passing the dicts explicitly is a better way.

What you're proposing sounds very unconventional. When code increases in size, you have to break down the code into a modular architecture, with clean APIs between modules. It also has to be something that is easy to comprehend, easy to explain, and easy to hand over to another programmer to modify/improve/debug it. What you're proposing sounds like it is not a clean API, unconventional, with a non-obvious data flow. I suspect it would probably make many programmers grumpy when they saw it. :)

Another option would be to make the functions members of a class, with the data being in the class instance. That could work well if your problem can be modelled as several functions operating on the data object.