Is there a way to pass flags to runpy?

Question

I am looking for a way to pass miscellaneous options using runpy or other tools.

In particular, I would like to get the output of an optimized python script in another non-optimized python script.

python -O tobeoptimized.py

I have tried using subprocess but I can't extract the object that I need as I do in runpy.

from subprocess import PIPE, run
command = ['python','-O','tobeoptimized.py']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)

@NizamMohamed I use this example = runpy.run_path(mypath) example[resourcetype].dosomething() — G M, Jul 20 '21 at 06:18

Peter Badida · Answer 1 · 2021-07-14T14:20:02.300

6

Not with runpy because it isn't spawning a new process, (exec()). If it could have been done this way, this question would have had an answer. So we're pretty much falling into the C codebase or extension world where the core of CPython is modified in such a manner it can be accessed at least from C API if not from Python API.

Instead try using subprocess for passing the flags. The output can be adjusted.

If the problem is that you want to extract an object out of the process (thus subprocess "failing") you need to look into the pickle module or simply drop it to a file/buffer/socket and retrieve the data + reassemble in a custom manner.

Subprocess isn't really mandatory, however if there were any other implementation, it won't "hack it away" because it's the compiled core that's in the way, not something that can be monkey-patched away.

I guess we need a new feature from the core devs? :)

Regarding the subprocess output, this works for me just fine.

# hello.py
print("Hello")

# main.py
from subprocess import Popen, PIPE

process = Popen(
    ["python", "-O", "hello.py"],
    stdout=PIPE, stderr=PIPE, universal_newlines=True
)
print(process.stdout.read())

edited Jul 14 '21 at 14:20

answered Jul 14 '21 at 13:51

Peter Badida

11,310
10
44
90

1

Thanks for this nice answer, yes I will have to find another solution, it's a pity I find very useful optimize some part of the code for some application where I am sure no errors will rise from assertions. Would be nice to have this feature. – G M Jul 14 '21 at 14:28
Same here and mainly the disabling part dynamically would be beneficial so it's all kept like the full ecosystem is i.e. "everything is a dictionary" and hacking away the whole language because you can. Nevertheless, there's a hope that in the future, perhaps in Python 4.x (major core rework :P ), it'll be possible. – Peter Badida Jul 14 '21 at 14:31
@GM: I'm curious -- how many assertions does your code have that disabling them provides a significant speedup? Or is that the assertions you do have are expensive to calculate? – Ethan Furman Jul 20 '21 at 21:08
@EthanFurman Imagine code with [`ensure`](https://github.com/kislyuk/ensure) for development + testing and `-O2` for production. – Peter Badida Jul 20 '21 at 21:55
@PeterBadida: Ouch. `ensure` is way too verbose for me. I will also note that `ensure` cannot be turned off with `-O` nor with `-OO`. – Ethan Furman Jul 20 '21 at 22:26
1

@EthanFurman a lot, I also use __debug__ for some part of code the performance improvement in my machine are 1.19 seconds w/o optimization and with the optimization in 0.48 seconds so it is more than two times faster.. – G M Jul 21 '21 at 06:24

Nizam Mohamed · Accepted Answer · 2022-08-26T03:16:38.280

It's doable.
The compile built-in function takes optimize parameter. Its values are 0, 1 and 2. If it's 0 which is like without -O, 1 is -O and 2 is -OO on the command line.
To make runpy run module / path optimized it must be patched. Before doing that I'll define two functions for illustration.

This is a test module. If it's run without optimization it won't print.
app.py

assert 0, "assert zero"
print("Optimized")

These two functions don't do all the details as runpy does. So won't function in full.

import importlib
import runpy

optimize = 1


def run_module(mod_name):
    spec = importlib.util.find_spec(mod_name)
    source = spec.loader.get_source(spec.name)
    code = compile(source, spec.name + ".py", "exec", optimize=optimize)
    d = {}
    exec(code, d)
    return d


def run_path(path):
    with open(path) as f:
        source = f.read()
    code = compile(source, path, "exec", optimize=optimize)
    d = {}
    exec(code, d)
    return d

This one is to patch a function in runpy which runpy.run_module uses to get code object of the module to run. The patch provides optimized code object.

import runpy

optimize = 1

def _get_module_details_wrapper(func):
    def tmp(*args, **kwargs):
        mod_name, spec, _ = func(*args, **kwargs)
        source = spec.loader.get_source(spec.name)
        code = compile(source, spec.name + ".py", "exec", optimize=optimize)
        return mod_name, spec, code

    return tmp

runpy._get_module_details = _get_module_details_wrapper(runpy._get_module_details)
runpy.run_module('app')

UPDATE
runpy.run_path can be run with optimization turned on.

def optimize_compile(o):
    from functools import partial
    import builtins

    builtins.compile = partial(compile, optimize=o)


optimize_compile(1)
runpy.run_path("app.py")


optimize_compile(0)
try:
    runpy.run_path("app.py")
except AssertionError:
    print("assertion")

optimize_compile(1)
runpy.run_path("app.py")

This is a brilliant answer, with this method is probably also possible to pass some arguments to the source code before compiling. I am testing it right now. — G M, Jul 20 '21 at 06:34
Seems workable, but utilizes an external file. At this point one would rather utilize multiprocessing to have a clear interpreter's state instead of having it blow into face in the runtime somehow. Then again, perhaps one day I'll be forced to go for this hack, so... +1! :) — Peter Badida, Jul 20 '21 at 19:18

Is there a way to pass flags to runpy?

2 Answers2