Distributing set of python files as a single executable

Question

I have a python script, distributed across several files, all within a single directory. I would like to distribute this as a single executable file (for linux systems, mainly), such that the file can be moved around and copied easily.

I've come as far as renaming my main file to __main__.py, and zipping everything into myscript.zip, so now I can run python myscript.zip. But that's one step too short. I want to run this as ./myscript, without creating an alias or a wrapper script.

Is this possible at all? What's the best way? I thought that maybe the zip file could be embedded in a (ba)sh script, that passes it to python (but without creating a temporary file, if possible).

EDIT: After having another go at setuptools (I had not managed to make it work before), I could create sort of self-contained script, an "eggsecutable script". The trick is that in this case you cannot have your main module named __main__.py. Then there are still a couple of issues: the resulting script cannot be renamed and it still creates the __pycache__ directory when run. I solved these by modifying the shell-script code at the beginning of the file, and adding the -B flag to the python command there.

EDIT (2): Not that easy either, it was working because I still had my source .py files next to the "eggsecutable", move something and it stops working.

You might want to have a look at `setuptools`, more specifically [Automatic Script Creation](https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation). Other [Python packaging tools](http://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-setuptools-and-distutils2) have similar constructs. — dhke, Sep 21 '15 at 13:03

ntki · Answer 1 · 2015-09-22T07:59:51.063

1

You can edit the raw zip file as a binary and insert the shebang into the first line.

#!/usr/bin/env python
PK...rest of the zip

Of course you need an appropriate editor for this, that can handle binary files (e.g.: vim -b) or you can make it with a small bash script.

{ echo '#!/usr/bin/env python'; cat myscript.zip; } > myscript
chmod +x myscript
./myscript

edited Sep 22 '15 at 07:59

answered Sep 22 '15 at 07:47

ntki

2,149
1
16
19

That's of course easier than my solution. I tried this before and it didn't work, but that's because I didn't use the proper tool (`vim -b`). The problem is I don't know how to get the actual file name from within the script, as `__file__` gives the name within the zip. – Jellby Sep 22 '15 at 08:27
@Jellby `__file__` gives `./myscript/__main__.py` for me. You could extract the script name from that. Also `sys.argv` gives `['./myscript']`. – ntki Sep 22 '15 at 08:34
You could also the a look at what py.test does with `py.test --genscript=script`. It creates a self-contained script with the wrapped source in base64 inside of it. – ntki Sep 22 '15 at 08:38

score 0 · Answer 2 · answered Sep 21 '15 at 14:40

0

Firstly, there's the obligatory "thus isn't the way things are normally done, are you sure you want to do it THIS way?" warning. That said, to answer your question and not try to substitute it with what someone else thinks you should do...

You can write a script, and prepend it to a python egg. The script would extract the egg from itself, and obviously call exit before the egg file data is encountered. Egg files are importable but not executable, so the script would have to

Extract the egg from itself as an egg file with a known name in the current directory
Run python -m egg
Delete the file
Exit

Sorry, I'm on a phone at the moment, so I'll update with actual code later

answered Sep 21 '15 at 14:40

sirlark

2,187
2
18
28

Are you still on the phone? What is the normal way? – None Aug 16 '16 at 18:33
Distribution would normally be done with a setuptools created zip or tar.gz, or these days a wheel file. I was saying that distribution via executables is unusual in and of itself – sirlark Aug 16 '16 at 19:46
And yes, actually, still on a phone :) – sirlark Aug 16 '16 at 19:47

Jellby · Answer 3 · 2015-09-22T08:50:13.497

Continuing my own attempts, I concocted something that works for me:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import glob, os.path

from setuptools import setup

files = [os.path.splitext(x)[0] for x in glob.glob('*.py')]
thisfile = os.path.splitext(os.path.basename(__file__))[0]
files.remove(thisfile)

setup(
  script_args=['bdist_wheel', '-d', '.'],
  py_modules=files,
)

# Now try to make the package runnable:
# Add the self-running trick code to the top of the file,
# rename it and make it executable

import sys, os, stat

exe_name = 'my_module'
magic_code = '''#!/bin/sh
name=`readlink -f "$0"`
exec {0} -c "import sys, os; sys.path.insert(0, '$name'); from my_module import main; sys.exit(main(my_name='$name'))" "$@"
'''.format(sys.executable)

wheel = glob.glob('*.whl')[0]
with open(exe_name, 'wb') as new:
  new.write(bytes(magic_code, 'ascii'))
  with open(wheel, 'rb') as original:
    data = True
    while (data):
      data = original.read(4096)
      new.write(data)
os.remove(wheel)
st = os.stat(exe_name)
os.chmod(exe_name, st.st_mode | stat.S_IEXEC)

This creates a wheel with all *.py files in the current directory (except itself), and then adds the code to make it executable. exe_name is the final name of the file, and the from my_module import main; sys.exit(main(my_name='$name')) should be modified depending on each script, in my case I want to call the main method from my_module.py, which takes an argument my_name (the name of the actual file being run).

There's no guarantee this will run in a system different from the one it was created, but it is still useful to create a self-contained file from the sources (to be placed in ~/bin, for instance).

score 0 · Accepted Answer · answered Sep 22 '15 at 14:57

Another less hackish solution (I'm sorry for answering my own question twice, but this doesn't fit in a comment, and I think it is better in a separate box).

#!/usr/bin/env python3

modules = [
  [ 'my_aux', '''
def my_aux():
  return 7
'''],
  ['my_func', '''
from my_aux import my_aux
def my_func():
  print("and I'm my_func: {0}".format(my_aux()))
'''],
  ['my_script', '''
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from __future__ import (unicode_literals, division, absolute_import, print_function)
import sys
def main(my_name):
  import my_aux
  print("Hello, I'm my_script: {0}".format(my_name))
  print(my_aux.my_aux())
  import my_func
  my_func.my_func()
if (__name__ == '__main__'):
  sys.exit(main(__file__))
'''],
]

import sys, types
for m in modules:
  module = types.ModuleType(m[0])
  exec(m[1], module.__dict__)
  sys.modules[m[0]] = module
del modules

from my_script import main

main(__file__)

I think this is more clear, although probably less efficient. All the needed files are included as strings (they could be zipped and b64-encoded first, for space efficiency). Then they are imported as modules and the main method is run. Care should be taken to define the modules in the right order.

Distributing set of python files as a single executable

4 Answers4