why does a call to locals() add a reference?

Question

I don't understand the below behavior.

• How does locals() result in a new reference?
• Why doesn't gc.collect remove it? I didn't assign the result of locals() anywhere.

x

import gc

from sys import getrefcount

def trivial(x): return x

def demo(x):
    print getrefcount(x)
    x = trivial(x)
    print getrefcount(x)
    locals()
    print getrefcount(x)
    gc.collect()
    print getrefcount(x)


demo(object())

The output is:

$ python demo.py 
3
3
4
4

Answer 1

This has to do with 'fast locals' which are stored as a matched pair of tuples for fast integer indexing (one for names f->f_code->co_varnames, one for values f->f_localsplus). When locals() is called, the fast-locals are converted into a standard dict and tacked onto the frame structure. The relevant bits of the cpython code are below.

This is the implementing function for locals(). It does little more than call PyEval_GetLocals.

static PyObject *
builtin_locals(PyObject *self)
{
    PyObject *d;

    d = PyEval_GetLocals();
    Py_XINCREF(d);
    return d;
}   

In turn PyEval_GetLocals does little more than call PyFrame_FastToLocals.

PyObject *
PyEval_GetLocals(void)
{   
    PyFrameObject *current_frame = PyEval_GetFrame();
    if (current_frame == NULL)
        return NULL;
    PyFrame_FastToLocals(current_frame);
    return current_frame->f_locals;
}

This is the bit that allocates a plain-old dictionary for the frame's local variables and stuffs any "fast" variables into it. Since the new dict is tacked onto the frame structure (as f->f_locals), any "fast" variables get an extra reference upon a call to locals().

void
PyFrame_FastToLocals(PyFrameObject *f)
{
    /* Merge fast locals into f->f_locals */
    PyObject *locals, *map;
    PyObject **fast;
    PyObject *error_type, *error_value, *error_traceback;
    PyCodeObject *co;
    Py_ssize_t j;
    int ncells, nfreevars;
    if (f == NULL)
        return;
    locals = f->f_locals;
    if (locals == NULL) {
        /* This is the dict that holds the new, additional reference! */
        locals = f->f_locals = PyDict_New();  
        if (locals == NULL) {
            PyErr_Clear(); /* Can't report it :-( */
            return;
        }
    }
    co = f->f_code;
    map = co->co_varnames;
    if (!PyTuple_Check(map))
        return;
    PyErr_Fetch(&error_type, &error_value, &error_traceback);
    fast = f->f_localsplus;
    j = PyTuple_GET_SIZE(map);
    if (j > co->co_nlocals)
        j = co->co_nlocals;
    if (co->co_nlocals)
        map_to_dict(map, j, locals, fast, 0);
    ncells = PyTuple_GET_SIZE(co->co_cellvars);
    nfreevars = PyTuple_GET_SIZE(co->co_freevars);
    if (ncells || nfreevars) {
        map_to_dict(co->co_cellvars, ncells,
                    locals, fast + co->co_nlocals, 1);
        /* If the namespace is unoptimized, then one of the
           following cases applies:
           1. It does not contain free variables, because it
              uses import * or is a top-level namespace.
           2. It is a class namespace.
           We don't want to accidentally copy free variables
           into the locals dict used by the class.
        */
        if (co->co_flags & CO_OPTIMIZED) {
            map_to_dict(co->co_freevars, nfreevars,
                        locals, fast + co->co_nlocals + ncells, 1);
        }
    }
    PyErr_Restore(error_type, error_value, error_traceback);
}

Answer 2

I added a few prints to your demo code :

#! /usr/bin/python

import gc

from sys import getrefcount

def trivial(x): return x

def demo(x):
    print getrefcount(x)
    x = trivial(x)
    print getrefcount(x)
    print id(locals())
    print getrefcount(x)
    print gc.collect(), "collected"
    print id(locals())
    print getrefcount(x)


demo(object())

The output is then (on my machine):

3
3
12168320
4
0 collected
12168320
4

locals() actually creates a dict containing a ref on x, thus the ref inc. gc.collect() does not collect the locals dict, you can see it by printing the id, it's the same object returned twice, it is somehow memoized for this frame, thus not collected.

Answer 3

This is because locals() creates an actual dictionary and puts x inside it hence, increasing the reference count of x, this dictionary is probably cached.

So I changed the code by adding two lines

import gc

from sys import getrefcount

def trivial(x): return x

def demo(x):
   print getrefcount(x)
   x = trivial(x)
   print getrefcount(x)
   print "Before Locals ",  gc.get_referrers(x)
   locals()
   print "After Locals ",  gc.get_referrers(x)
   print getrefcount(x)
   gc.collect()
   print getrefcount(x)
   print "After garbage collect", gc.get_referrers(x)

demo(object())   

and here is the output of the code

3
3
Before Locals  [<frame object at 0x1f1ee30>]
After Locals  [<frame object at 0x1f1ee30>, {'x': <object object at 0x7f323f56a0c0>}]
4
4
After garbage collect [<frame object at 0x1f1ee30>, {'x': <object object at 0x7f323f56a0c0>}]

Seems like it is caching the dict value even after garbage collection for future calls to locals().