How can I use inspect.getsource from within a dynamically executed string of code?

Question

If I have this code in a file:

import inspect

def sample(p1):
    print(p1)
    return 1

print(inspect.getsource(sample))

When I run the script, it works as expected: on the last line, the source code for the sample function is printed.

However, if I create a string with that same code, and supply it to exec, like so:

source = """import inspect

def sample(p1):
    print(p1)
    return 1

print(inspect.getsource(sample))
"""

exec(source)

This fails with the message OSError: could not get source code.

How can I make this work while using exec to run the code - with the call to inspect.getsource being part of the dynamically-executed code string?

Answer 1

Regrettably, as I'm sure you've realised, this simply is not possible using exec...but it is possible using exec_module.

If you are not required to use exec specifically, this may work for you:

import tempfile
from importlib import util
import importlib.machinery

raw = """
def sample(p1):
    print(p1)
    return 1

import inspect
print(inspect.getsource(sample))
"""

# Create a temporary source code file
with tempfile.NamedTemporaryFile(suffix='.py') as tmp:
    tmp.write(raw.encode())
    tmp.flush()

    # Now load that file as a module
    spec = util.spec_from_file_location('tmp', tmp.name)
    module = util.module_from_spec(spec)
    spec.loader.exec_module(module)

    # ...or, while the tmp file exists, you can query it externally
    import inspect
    print(inspect.getsource(module.sample))

The TLDR is, without a source file, it is just flat out not-possible to use getsource to read the source code of a function, because the implementation reads the source code from the file defined in f.__code__.co_filename on the function (read-only, so monkey patching is impossible).

However, if you write the source code to a temporary file, you can load that file as a module (exec_module is practically identical to the effect of exec, you just have to do a few other steps first).

Note: tmp.flush() <-- don't forget this, tmp.write() doesn't write until you flush, and you'll be 'loading' an empty file if you don't flush it first.

Answer 2

If your code is like this:

exec("""\
import inspect

def sample(p1):
    print(p1)
    return 1

print(inspect.getsource(sample))\
""")

And the exception is OSError: could not get source code then that probably means that inspect.getsource() requires a file's source to inspect. exec dynamically executes Python code so the sample function that's being defined from exec is not stored in a file.

Something like this:

def sample(p1):
    print(p1)
    return 1

exec("""\
import inspect
print(inspect.getsource(sample))\
""")

would work because the function sample is defined in the source code.

Answer 3

Actually, you can solve this issue without using temporary files.

Apparently, this exact issue was encountered during the development of the builtin doctest module. This is even mentioned in the source code of inspect. It turns out that reading source files is done through the linecache.getlines function. What doctest does it that it swaps that function for its own wrapper. Based on this, we can create an analogous patch for exec:

def better_exec(code_, globals_=None, locals_=None, /, *, closure=None):
    import ast
    import linecache

    if not hasattr(better_exec, "saved_sources"):
        old_getlines = linecache.getlines
        better_exec.saved_sources = []

        def patched_getlines(filename, module_globals=None):
            if "<exec#" in filename:
                index = int(filename.split("#")[1].split(">")[0])
                return better_exec.saved_sources[index].splitlines(True)
            else:
                return old_getlines(filename, module_globals)

        linecache.getlines = patched_getlines
    better_exec.saved_sources.append(code_)
    exec(
        compile(
            ast.parse(code_),
            filename=f"<exec#{len(better_exec.saved_sources) - 1}>",
            mode="exec",
        ),
        globals_,
        locals_,
        closure=None,
    )

It works even if you try to get the source code from within the exec:

>>> namespace_ = {}
>>> better_exec("""\
... def foo():
...     print("foo")
... import inspect
... foo_source = inspect.getsource(foo)
... """, namespace_)
>>> print(namespace_["foo_source"])
def foo():
    print("foo")
>>> namespace_["foo"]()
foo

Each call to exec stores its input in the better_exec.saved_sources array. As such, if there are many calls to it, this could potentially cause high memory usage. This could be solved, for example, by swapping some of these sources to disk or simply deleting them.

If you are getting an OSError: source code not available from within inspect, make sure that your filename (<exec#...> here) begins with < and ends with >.

Note that while this successfully exposes the code to inspect, and pdb, it is, unfortunately, not visible in native exception stack traces:

>>> namespace_ = {}
>>> better_exec("""\
... def f():
...     raise RuntimeError()
... """, namespace_)
>>> import inspect
>>> print(inspect.getsource(namespace_["f"])) # Works!
def f():
    raise RuntimeError()
>>> namespace_["f"]()
Traceback (most recent call last):
  File "/app/output.s", line 37, in <module>
    namespace_["f"]() # No source :(
    ^^^^^^^^^^^^^^^^^
  File "<exec#0>", line 2, in f
RuntimeError

As as sidenote, if you are using VSCode, then by disabling Just My Code, you will even be able to step through the code given to exec: