Python program doesn't write to output csv, everything else seems to work correctly

Question

    from subprocess import check_output
import csv, operator


extinction_pct = operator.itemgetter('AOT 500','AOT 675','AOT 870','AOT 936','AOT 1020')

with open('csv_export.csv') as f_csv:
    reader = csv.DictReader(f_csv)
    for row in reader:
        with open("INPUT", 'w') as f_in:
             f_in.write("&INPUT\n")
             f_in.write("WLINF = 0.250\n")               #lower frequency value
             f_in.write("WLSUP = 4.0\n")                 #highest frequency value
             f_in.write("WLINC = 0.5\n")                     #wavelength increment
             f_in.write("IDAY = 289\n")                  #computing for a specific day
             #f_in.write("ALAT = {Lat}\n".format(**row))    # for Python versions less than 3.6
             f_in.write(f"ALAT = {row['Lat']}\n")          #latitude of the location
         #f_in.write("ALON = {Long}\n".format(**row))    # for Python versions less than 3.6
             f_in.write(f"ALON = {row['Long']}\n")          #longitude of the location
             f_in.write("IDATM = 3\n")                   #atmopsheric model 2 - mid latitude summer
             f_in.write("ISALB = 5\n")                         #surface albedo feature
             f_in.write("IAER = 5\n")                          #boundary layer aerosol type selection - 5 - user defined spectral dependance of BLA
             f_in.write("WLBAER = .500,.675,.870,.936,1.02\n") #wavelenght points for IAER
             f_in.write("WBAER = 5*0.9\n")                      #single scattering albedo
             f_in.write("GBAER = 5*0.8\n")                      #assymetric factor used with IAER
         #f_in.write("TIME = {sama]}\n".format(**row))    # for Python versions less than 3.6
             f_in.write(f"TIME = {row['sama']}\n")                       #Time in IST format (-5.30hr)
        #f_in.write("QBAER = {}\n".format(','.join(extinction_pct(row)))    # for Python versions less than 3.6
             f_in.write(f"QBAER = {','.join(extinction_pct(row))}\n") #extinction efficiency percentage
             f_in.write("ZOUT = 0.0,15.0\n")                         #TOA defining
             f_in.write("/\n")
          check_output('sbdart >> output1.csv',shell=True)  #slarrt is the program, and ouytput.csv is the output file

This is my code, with help from @wwii

My last line, check_output csv doesnt write to my output file at all. What could be the issue? thanks

sbdart is a program, that takes the INPUT file and outputs in the command line

try `check_output("sbdart", ">", "output1.csv",shell=True)` should be a single `>` — Siddharth Chabra, Jun 24 '18 at 16:24
are you sure the `sbdart >> output1.csv` is working in the command prompt? — Siddharth Chabra, Jun 24 '18 at 16:34
How are you creating the file `output1.csv`, with this `with open("INPUT", 'w') as f_in:`? Surely that is creating a file called `INPUT`. — Rolf of Saxony, Jun 24 '18 at 17:21
Yes, it is indeed. This code 1. Creates a file called INPUT from the csv data csv_export.csv, one for each row of the sheet 2. It is then *supposed* to pass that to sbdart, and take the output that is generated in the terminal and copy that to a third file, output.csv — samhain, Jun 24 '18 at 17:24
Then I suggest that you take a hard look at the `sbdart` program, of which we have zero knowledge. — Rolf of Saxony, Jun 24 '18 at 17:26
The program works by itself, and in cmd when i directly type the sbdart >> output.csv, it works perfectly would you like a copy of the program? It has no configuration options or even an interface, hence the confusion. Please help me out, im at my wits end — samhain, Jun 24 '18 at 17:28
change to `check_output(['sbdart >> output1.csv'],shell=True)` note [...] and see if it makes a difference — Rolf of Saxony, Jun 24 '18 at 17:34

Siddharth Chabra · Answer 1 · 2018-06-24T16:43:53.970

0

Using method provided here you can try using this.

import subprocess
proc = subprocess.Popen('cmd.exe', stdin = subprocess.PIPE, stdout = subprocess.PIPE)
stdout, stderr = proc.communicate('sbdart >> output.csv')

Make sure you put the full path of sbdart or navigate to the folder having sbdart or add location of sbdart to system path

There are a bunch of other methods in the link provided

edited Jun 24 '18 at 16:43

answered Jun 24 '18 at 16:28

Siddharth Chabra

448
6
22

replace check_output line with these 2 lines – Siddharth Chabra Jun 24 '18 at 16:46
I honestly suspect its a path problem, in your original code put full path of sbdart with extension and it should work – Siddharth Chabra Jun 24 '18 at 16:48
That somehow breaks the whole code, now even INPUT isn't generated properly – samhain Jun 24 '18 at 16:48
in your original code replace sbdart with full file path with extension does it work? – Siddharth Chabra Jun 24 '18 at 16:51
cant help further without understanding sbdart – Siddharth Chabra Jun 24 '18 at 16:52
How can i upload you a copy? – samhain Jun 24 '18 at 17:00
the program itself, just takes the INPUT as input, and provides output directly in command line, to be copied to the csv. – samhain Jun 24 '18 at 17:09
like I said it should work. I still think it is a path problem – Siddharth Chabra Jun 25 '18 at 12:24

score 0 · Answer 2 · answered Jun 24 '18 at 17:54

Working on Linux with python 3.5
Assume sbdart is executable and we have a file called output1.csv
sbdart looks like this for our test case:

echo $1
echo "$(cat $1)"

output1.csv is as follows:

&INPUT
WLINF = 0.250
WLSUP = 4.0
WLINC = 0.5
IDAY = 289
ALAT = {row['Lat']}
ALON = {row['Long']}
IDATM = 3
ISALB = 5
IAER = 5
WLBAER = .500,.675,.870,.936,1.02
WBAER = 5*0.9
GBAER = 5*0.8
TIME = {row['sama']}
QBAER = {','.join(extinction_pct(row))}
ZOUT = 0.0,15.0
/


>>> import subprocess
>>> subprocess.check_output(['./sbdart output1.csv'],shell=True)
b"output1.csv\n&INPUT\nWLINF = 0.250\nWLSUP = 4.0\nWLINC = 0.5\nIDAY = 289\nALAT = {row['Lat']}\nALON = {row['Long']}\nIDATM = 3\nISALB = 5\nIAER = 5\nWLBAER = .500,.675,.870,.936,1.02\nWBAER = 5*0.9\nGBAER = 5*0.8\nTIME = {row['sama']}\nQBAER = {','.join(extinction_pct(row))}\nZOUT = 0.0,15.0\n/\n"
>>>

Python program doesn't write to output csv, everything else seems to work correctly

2 Answers2

Linked