Python subprocess call precompiled java

Question

This works from the windows command line:

c:\mallet\bin\mallet run

I've tried

subprocess.call(['c:\mallet\bin\mallet', 'run'])

and get an error

WindowsError: [Error 2] The system cannot find the file specified

and I've tried

subprocess.call(['c:/mallet/bin/mallet', 'run'])

and get the error

WindowsError: [Error 193] %1 is not a valid Win32 application

What do I have to pass to subprocess.call()?

For completeness sake the complete command I would like to pass is:

bin\mallet run cc.mallet.topics.tui.DMRLoader texts.txt features.txt instance.mallet

My vague idea is that this is a precompiled java class that I'm calling somehow, but I don't really understand what I'm doing here.

Here are the two mallet-files in the folder bin:

mallet.bat

@echo off

rem This batch file serves as a wrapper for several
rem  MALLET command line tools.

if not "%MALLET_HOME%" == "" goto gotMalletHome

echo MALLET requires an environment variable MALLET_HOME.
goto :eof

:gotMalletHome

set MALLET_CLASSPATH=%MALLET_HOME%\class;%MALLET_HOME%\lib\mallet-deps.jar
set MALLET_MEMORY=1G
set MALLET_ENCODING=UTF-8

set CMD=%1
shift

set CLASS=
if "%CMD%"=="import-dir" set CLASS=cc.mallet.classify.tui.Text2Vectors
if "%CMD%"=="import-file" set CLASS=cc.mallet.classify.tui.Csv2Vectors
if "%CMD%"=="import-smvlight" set CLASS=cc.mallet.classify.tui.SvmLight2Vectors
if "%CMD%"=="train-classifier" set CLASS=cc.mallet.classify.tui.Vectors2Classify
if "%CMD%"=="train-topics" set CLASS=cc.mallet.topics.tui.Vectors2Topics
if "%CMD%"=="infer-topics" set CLASS=cc.mallet.topics.tui.InferTopics
if "%CMD%"=="estimate-topics" set CLASS=cc.mallet.topics.tui.EstimateTopics
if "%CMD%"=="hlda" set CLASS=cc.mallet.topics.tui.HierarchicalLDATUI
if "%CMD%"=="prune" set CLASS=cc.mallet.classify.tui.Vectors2Vectors
if "%CMD%"=="split" set CLASS=cc.mallet.classify.tui.Vectors2Vectors
if "%CMD%"=="bulk-load" set CLASS=cc.mallet.util.BulkLoader
if "%CMD%"=="run" set CLASS=%1 & shift

if not "%CLASS%" == "" goto gotClass

echo Mallet 2.0 commands: 
echo   import-dir        load the contents of a directory into mallet instances (one per file)
echo   import-file       load a single file into mallet instances (one per line)
echo   import-svmlight   load a single SVMLight format data file into mallet instances (one per line)
echo   train-classifier  train a classifier from Mallet data files
echo   train-topics      train a topic model from Mallet data files
echo   infer-topics      use a trained topic model to infer topics for new documents
echo   estimate-topics   estimate the probability of new documents given a trained model
echo   hlda              train a topic model using Hierarchical LDA
echo   prune             remove features based on frequency or information gain
echo   split             divide data into testing, training, and validation portions
echo Include --help with any option for more information


goto :eof

:gotClass

set MALLET_ARGS=

:getArg

if "%1"=="" goto run
set MALLET_ARGS=%MALLET_ARGS% %1
shift
goto getArg

:run

java -Xmx%MALLET_MEMORY% -ea -Dfile.encoding=%MALLET_ENCODING% -classpath %MALLET_CLASSPATH% %CLASS% %MALLET_ARGS%

:eof

and mallet

#!/bin/bash


malletdir=`dirname $0`
malletdir=`dirname $malletdir`

cp=$malletdir/class:$malletdir/lib/mallet-deps.jar:$CLASSPATH
#echo $cp

MEMORY=1g

JAVA_COMMAND="java -Xmx$MEMORY -ea -Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -classpath $cp"

CMD=$1
shift

help()
{
cat <<EOF
Mallet 2.0 commands: 

  import-dir         load the contents of a directory into mallet instances (one per file)
  import-file        load a single file into mallet instances (one per line)
  import-svmlight    load SVMLight format data files into Mallet instances
  train-classifier   train a classifier from Mallet data files
  classify-dir       classify data from a single file with a saved classifier
  classify-file      classify the contents of a directory with a saved classifier
  classify-svmlight  classify data from a single file in SVMLight format
  train-topics       train a topic model from Mallet data files
  infer-topics       use a trained topic model to infer topics for new documents
  evaluate-topics    estimate the probability of new documents under a trained model
  hlda               train a topic model using Hierarchical LDA
  prune              remove features based on frequency or information gain
  split              divide data into testing, training, and validation portions

Include --help with any option for more information
EOF
}

CLASS=

case $CMD in
    import-dir) CLASS=cc.mallet.classify.tui.Text2Vectors;;
    import-file) CLASS=cc.mallet.classify.tui.Csv2Vectors;;
        import-svmlight) CLASS=cc.mallet.classify.tui.SvmLight2Vectors;;
    train-classifier) CLASS=cc.mallet.classify.tui.Vectors2Classify;;
        classify-dir) CLASS=cc.mallet.classify.tui.Text2Classify;;
        classify-file) CLASS=cc.mallet.classify.tui.Csv2Classify;;
        classify-svmlight) CLASS=cc.mallet.classify.tui.SvmLight2Classify;;
    train-topics) CLASS=cc.mallet.topics.tui.Vectors2Topics;;
    infer-topics) CLASS=cc.mallet.topics.tui.InferTopics;;
    evaluate-topics) CLASS=cc.mallet.topics.tui.EvaluateTopics;;
    hlda) CLASS=cc.mallet.topics.tui.HierarchicalLDATUI;;
    prune) CLASS=cc.mallet.classify.tui.Vectors2Vectors;;
    split) CLASS=cc.mallet.classify.tui.Vectors2Vectors;;
    bulk-load) CLASS=cc.mallet.util.BulkLoader;;
    run) CLASS=$1; shift;;
    *) echo "Unrecognized command: $CMD"; help; exit 1;;
esac

$JAVA_COMMAND $CLASS $*

I typed `assoc c:\mallet\bin\mallet` to windows command line and the return was `File association not found for extension c:\mallet\bin\mallet` I tried it also with `/` with same results. — Artturi Björk, Mar 27 '15 at 21:19
The problem is that `c:\mallet\bin\mallet` is not exceutable, what exactly is `c:\mallet\bin\mallet`? If it is a java file why not run it with java? — Padraic Cunningham, Mar 27 '15 at 21:28
Unfortunately I don't know. There are two files in the folder named mallet: mallet.bat (begins `@echo off`) and mallet (begins `#!/bin/bash`). — Artturi Björk, Mar 27 '15 at 21:34
Have you tried running `subprocess.call(['c:/mallet/bin/mallet.bat', 'run'])` ? Or `subprocess.call(['cmd', 'c:/mallet/bin/mallet.bat', 'run'])` ? — Sylvain Leroux, Mar 27 '15 at 21:55
I had not, but now that I have I can report that they both work as expected. Thanks! — Artturi Björk, Mar 27 '15 at 22:09
@Artturi I've posted that as an answer, with some explanations (I'm not a regular Windows user -- so I didn't get too much into details, though). Please note in your case, the error message was rather confusing because you had in fact a `mallet` (without extension) file in the same folder. But it was a bash (i.e. Unix shell) executable -- so not understood by "Windows". — Sylvain Leroux, Mar 27 '15 at 22:17

Sylvain Leroux · Accepted Answer · 2015-03-30T09:50:59.440

3

When you call a program without extension, the Windows shell will try several standard extensions (.BAT, .EXE, ...) in order to guess the file you are trying to call.

If you want to execute your program without a shell to perform that look-up phase, you need to pass the full name of the batch you are trying to execute -- incl. the .BAT extension:

subprocess.call(['c:/mallet/bin/mallet.bat', 'run'])

edited Mar 30 '15 at 09:50

answered Mar 27 '15 at 22:15

Sylvain Leroux

50,096
7
103
125

1

related: [on the difference between `CreateProcess` (default) and `cmd.exe` (`shell=True`) command lookup](http://stackoverflow.com/a/25167402/4279). – jfs Mar 30 '15 at 18:50

Zizouz212 · Answer 2 · 2015-03-27T21:38:45.833

0

Make sure that you pass the shell = True argument to subprocess.call(). However, it poses security issues, so make sure you look at the documentation and understand how it works.

subprocess.call(['c:/mallet/bin/mallet', 'run'], shell = True)

Also, when using strings to identify paths that contain backslashes, make it a raw string (r"This is a raw string!"), so that it doesn't implement anything else (such as newline tokens).

If my above suggestions don't work, there are only two things that I can thing of:

The file you are trying to execute may not be an application file (.exe file). I don't work with Windows so I'm not sure about this one, but it could likely be a possibility.
Something in the file(s) is broken or something like that.

subprocess.call() docs

subprocess.call() Security Issues

edited Mar 27 '15 at 21:38

answered Mar 27 '15 at 21:02

Zizouz212

4,908
5
42
66

Thanks for the suggestion. Do you have any idea what kind of security issues using shell=True poses? I think it quite unlikely that mallet would be a security risk. I tried the `r` in front of the string, but got the same `WindowsError: [Error 193] %1 is not a valid Win32 application` error. – Artturi Björk Mar 27 '15 at 21:28
The only security issues that `shell = True` imposes is if commands are based from external input. I'll set a link to the docs. I doubt you'll have to worry about it in this case anyway though ;) – Zizouz212 Mar 27 '15 at 21:31
Your answer works and I'm going to accept it in a day or two, but would just like to hear if anyone has a suggestion how to get the `subprocess.call()` to work without `shell = True`. – Artturi Björk Mar 27 '15 at 21:51
Sure no problem. I should probably point out that you will always need `shell = True`. In other functions, such as `os.system`, it's like processing `subprocess.call` with the `shell = True` argument. – Zizouz212 Mar 27 '15 at 21:53

score -1 · Answer 3 · answered Mar 27 '15 at 21:02

-1

Well, maybe the problem is with the backslashes.

From their docs:

The backslash (\) character is used to escape characters that otherwise have a special meaning, such as newline, backslash itself, or the quote character.

So you probably should do:

subprocess.call(['c:\\mallet\\bin\\mallet', 'run'])

answered Mar 27 '15 at 21:02

brunodea

220
1
10

OP notes that they tried forward slashes, which are interpreted the same way as backslashes and don't need to be escaped. – jedwards Mar 27 '15 at 21:05
Thanks for the suggestion. I tried and got the `WindowsError: [Error 193] %1 is not a valid Win32 application` error. – Artturi Björk Mar 27 '15 at 21:26

Python subprocess call precompiled java

3 Answers3