0

Following on from my previous question which relates to a different problem, I thought I would frame in terms of this problem.

NOTE: I have tried all versions of subprocess.Popen(), communicate(), wait(), such as:

vowpal = subprocess.Popen('../../vowpal.sh',shell=True, stdout=subprocess.PIPE)
vowpal.communicate()
print vowpal.returncode

I have a file structure like this:

├── src
│   ├── main
│   │   ├── costSensitiveClassifier.py
└── vowpal.sh
|
├── data
│   ├── output
│   │   ├── cost
|   |   |_______openCostClassifier.dat
|   |   |

And within costSensitiveClassifier.py, I essentially am trying to run a script called vowpal.sh that does some manipulations on openCostClassifer.dat and outputs some files into the same folder as that file.

The code within costSensitiveClassifier.py is:

import subprocess
print "Starting cost sensitive predictions using batch script\n"
subprocess.call("../../vowpal.sh")
print "Ending predictions"

And the code within vowpal.sh is:

# !/bin/bash
vw --csoaa 24 data/output/cost/openCostClassifier.dat -f data/output/cost/csoaa.model
vw -t -i data/output/cost/csoaa.model data/output/cost/openCostClassifier.dat -p data/output/cost/csoaa.predict

The issue is always that the second line of the bash script needs there to be something called csoaa.model to be output first (see line 1 of the bash script), and once finished, the second line should run which uses that file to output csoaa.predict. However, this is not the case and I get when running the python file I get in my error code related to the bash script:

vw (./io_buf.h:123): can't open: data/output/cost/csoaa.modelerrno = No such file or directory

I found this link but not sure if it is what I need (should I adapt the sleep to be the expected time for the bash script to finish and create the files I need to work on?).

Community
  • 1
  • 1
Dhruv Ghulati
  • 2,976
  • 3
  • 35
  • 51
  • It seems like you need a work flow management library like [snakemake](https://bitbucket.org/snakemake/snakemake/wiki/Home) or [ruffus](http://www.ruffus.org.uk/) – Chris_Rands Aug 04 '16 at 08:17
  • Also your error suggests your script is looking for `csoaa.modelerrno` not `csoaa.model` – Chris_Rands Aug 04 '16 at 08:32

1 Answers1

1

This is a problem in vowpal. It tries to create an error file, which it can't. One of the reasons could be that the directory "data/output/cost/" doesn't exist.

That's what was causing same error for me.

Slaz
  • 11
  • 1