How to make python package refer to packaged, non-python executable (.exe); Windows

Question

I am putting together a Python package that uses a Cygwin executable without having Cygwin installed. This means that I have an executable (.exe) file and a library (.dll) file inside my package. I am doing this so that the tool can be used on Windows by people who only use Windows and Python. I am new to Python packages, so I appreciate any help.

---

The Main Question

How do I make my package point to the executable file inside my package? This would be used, for example, instead of the executable referenced by PATH. The answer might be out there, but all I've found in my search is a bunch of info on how to create an executable from a Python script, which is NOT what I want.

---

Details/My Attempts

The tool is sclite, a tool to score speech recognition output. For information about how I get the tool available on Windows, have a look at the How to Install SCLITE (for reproduce-ability) section, below.

Here's a small, "toy" example of how I have my package set up.

$ tree package_holder_dir/
package_holder_dir/
├── bbd_package
│   ├── __init__.py
│   ├── non_py_exe_dll
│   │   ├── cygwin1.dll
│   │   └── sclite.exe
│   ├── score
│   │   ├── __init__.py
│   │   └── run_sclite.py
│   └── score_transcript.py
├── MANIFEST.in
├── README.txt
└── setup.py

3 directories, 9 files

My first guess was that the sclite.exe should go in the MANIFEST.in file.

# @file MANIFEST.in

include ./README.txt
include ./bbd_package/non_py_exe_dll/sclite.exe
include ./bbd_package/non_py_exe_dll/cygwin1.dll

I also tried putting them in setup.py

My setup.py is as follows

#!/usr/bin/env/python3
# -*- coding: utf-8 -*-
# @file setup.py
    
import setuptools
from distutils.core import setup

with open ("README.txt", "r") as fh:
  long_description = fh.read()

setup(
  name='bbd_package',
  url='user@host.com',
  author='bballdave025',
  author_email='not.likely@idontwantspam.com',
  packages=setuptools.find_packages(),
  #data_files=[('lib', ['./non_py_exe_dll/cygwin1.dll']),
  #                     './non_py_exe_dll/sclite.exe'],
  version='0.0.1',
  description="Example for SO",
  long_description=long_description,
  include_package_data=True
) ##endof:  setup

Note this source (archived) for my use of data_files for the location of the DLL. However, I could not find the files when I installed the distribution elsewhere. That's why they are commented out here.

Using MANIFEST.in seemed to work, but then I had to use a relative path to access the executable. That won't work when trying to import bbd_package in another directory.

Let me try to illustrate with my two Python files:

score_transcript.py simply calls run_sclite.py.

#!/usr/env/bin python3
# -*- coding: utf-8 -*-
# @file run_sclite.py

import os, sys, subprocess

def run_sclite(hyp, ref):
  subprocess.call(['../non_py_exe_dll/sclite.exe', '-h', hyp, '-r', ref, '-i', 'rm', \
                   '-o', 'all snt'])

I can install it on my system:

C:\toy_executable_example\package_holder_dir>pip install .

Then if I happen to be in the directory with run_sclite.py

C:\toy_executable_example\package_holder_dir\bbd_package\score>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bbd_package.score.run_sclite
>>> bbd_package.score.run_sclite.run_sclite('a.hyp', 'a.ref')
sclite Version: 2.10, SCTK Version: 1.3
...output that shows it works...
>>>

However, from any other directory, no dice.

C:\Users\me>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bbd_package.score.run_sclite
>>> bbd_package.score.run_sclite.run_sclite('a.hyp', 'a.ref')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\site-packages\bbd_package\score\run_sclite.py", line 9, in run_sclite
    '-o', 'all snt'])
  File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Users\dblack\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>>

How can I tell Python to look for the executable inside my package?

---

System Details

This information is taken from running systeminfo from Window's Command Prompt.

OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.15063 N/A Build 15063
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Workstation
OS Build Type:             Multiprocessor Free

---

How to Install SCLITE (for reproduce-ability)

I'm including a link to the installation instructions (using Cygwin) here, look for my comment. Here are the commands with no explanation

$ cd
$ git clone https://github.com/kaldi-asr/kaldi.git
$ cd kaldi/tools
$ extras/check_dependencies.sh
$ make -j $(nproc --all)
$ cp -R sctk-2.4.10 ~/
$ cd
$ rm -rf kaldi
$ cd sctk-2.4.10/
$ cp $HOME/.bashrc "${HOME}/.bashrc.$(date +%Y%m%d-%H%M%S).bak"
$ echo -e "\n\n## Allow access to sclite, rfilter, etc" >> $HOME/.bashrc
$ echo 'export PATH='"$(pwd)/bin"':$PATH' >> $HOME/.bashrc
$ source ~/.bashrc

I'm also including a link (archived) to "quicker" instructions and the information about the EXE and DLL files. This is all thanks to @benreaves

Easier (but even uglier) workaround, which I gave to an intern who did some Word Error Rate calculations for me:

On my laptop with Cygwin installed, I create sclite.exe using my

cp src/*/*.c . ; make all

workaround from my previous message

I create a zip file containing sclite.exe and cygwin1.dll from my laptop's c:/cygwin/bin/ folder

Email that zip file to her, and she copies both files in a single new folder on her laptop and set PATH=.;%PATH%

I don't set the PATH, because I want this to be distributible.

This worked just fine on her laptop running Windows 7, and she didn't need Cygwin or any other NIST software on her laptop.

-Ben

Answer 1

I finally got this to work. The sources which I patched together are below. The basic answer is that I used the __file__ variable to find the directory from which the packaged module was called, then used relative paths to get to my executable. I'm not really satisfied with this solution (here {archived} are some situations in which it won't work), but it gets the job done for now.

Specifics

The MANIFEST.in file stayed the same:

# @file MANIFEST.in
# @author bballdave025

include ./README.txt
include ./bbd_package/non_py_exe_dll/sclite.exe
include ./bbd_package/non_py_exe_dll/cygwin1.dll

However, I needed to add the following to the exe-running code. This is in run_sclite.py

from pathlib import Path
#...
path_to_here = os.path.dirname(os.path.abspath(__file__))
path_to_pkg_root = Path(path_to_here).resolve().parents[1]
path_to_exe = os.path.join(path_to_pkg_root, 'non_py_exe_dll')
#...

Note that Python version >= 3.4 is necessary for pathlib

Here is the whole (run_sclite.py) file:

#!/usr/env/bin python3
# -*- coding: utf-8 -*-
# @file run_sclite.py
# @author bballdave025

import os, sys, subprocess
from pathlib import Path

def run_sclite(hyp, ref):
  path_to_here = os.path.dirname(os.path.abspath(__file__))
  path_to_pkg_root = Path(path_to_here).resolve().parents[1]
  path_to_exe = os.path.join(path_to_pkg_root, 'non_py_exe_dll')
  
  subprocess.call([os.path.join(path_to_exe, 'sclite.exe'), '-h', hyp, '-r', ref, \
                   '-i', 'rm', '-o', 'all snt'])
  
##endof:  run_sclite(hyp, ref)

---

Sources

Mouse vs. Python, with clarification on when this solution won't work and other alternatives. (archived)

Getting to package root directory (SO) (archived ; Look for "Using os.path").

What finally got me to the answer: Python Packaging Tutorial . Look at the 'Note:'. (archived)