Wait for external program to finish writing to file

Question

I have a Python script that calls an external program (sox to be precise). Now I have to do several things with sox, but always have to wait until one file is done writing so I can use it as an input file in my next command.

subprocess.wait() doesn't work, because the execution of sox will be done, but the file won't be done writing.

Here is the code I have:

import tempfile
import shlex

file_url = '/some/file.wav'
out_file = 'some/out.wav'
tmp_file = tempfile.NamedTemporaryFile()
pad_cmd = 'sox ' + file_url + ' ' + tmp_file.name + ' pad 0.0 3.0'
subprocess.call(shlex.split(pad_cmd))
trim_cmd = 'sox ' + tmp_file.name + ' -t wav ' + out_file + ' trim 0.0 3.0'

Any way to wait for the file to finish writing to tmp_file.name without using a timer that waits for a fixed amount of time. Or is there a built-in way in sox to use the output file as input? The special filename - didn't work for pad and trim.

*"the execution of sox will be done, but the file won't be done writing."* -- if it is true (it might be if `sox` is just a launcher for another program that does the actual writing to the file e.g., if the file is opened in a tab of already existing text editor) then in that case: what you ask **can't be done** for an arbitrary subprocess, you might hack some solution specific to `sox` though. Are you absolutely sure that by the time `sox` exits the file is not done? — jfs, Mar 16 '14 at 09:17
Can you instruct `sox` to write to stdout instead of a file e.g., `sox input.wav - pad 0.0 3.0 | sox - -t wav out.wav trim 0.0 3.0`? — jfs, Mar 16 '14 at 09:26
@J.F.Sebastian Oh, didn't know about the special filenames. Unfortunately it doesn't seem to work though. I get `sox FAIL formats: can't determine type of `-'` errors for both `-` times. — Sennster, Mar 16 '14 at 09:35
@Sennster i had a similar issue but a different language.. http://stackoverflow.com/questions/22290316/how-to-wait-for-a-file-to-be-released-by-another-external-process i used a timer and check if the file is available to be moved (just a rename operation) if it fails(which means the file is not ready yet for any operation) I sleep then try again.. I do this `X` times before quitting my attempts.. You can do some testing to determine optimal wait times — Dexters, Mar 16 '14 at 09:43
@J.F.Sebastian I'm not absolutely sure, if by the time `sox` exits the file is done or not. Some tests of mine has suggested so. But I just found out that writing to `stdout` does work. You just have to specify the file type in both commands. Though `sox` has some sort of problem to create the header for the wav files, but that's not a problem in my case. The command should be `sox input.wav - -t wav pad 0.0 3.0 | sox - -t wav out.wav trim 0.0 3.0` — Sennster, Mar 16 '14 at 10:06
Oh. Little error in above command. It should be: `sox input.wav -t wav - pad 0.0 3.0 | sox -t wav - out.wav trim 0.0 3.0` — Sennster, Mar 16 '14 at 12:17

score 2 · Accepted Answer · edited May 23 '17 at 12:07

If sox works with stdin/stdout:

#!/bin/sh
sox input.wav -t wav - pad 0.0 3.0 | sox -t wav - out.wav trim 0.0 3.0

then you could write it in Python:

#!/usr/bin/env python
from subprocess import Popen, PIPE

pad = Popen(["sox", "input.wav", "-t", "wav", "-", "pad", "0.0", "3.0"],
            stdout=PIPE)
trim = Popen(['sox', '-t', 'wav', '-', 'out.wav', 'trim', '0.0', '3.0'],
             stdin=pad.stdout)
pad.stdout.close() # notify pad on write if trim dies prematurely
pipestatus = [p.wait() for p in [pad, trim]]

Note: if you don't use an untrusted input to construct the commands then you could pass them as a single line for readability (the shell creates the pipe in this case):

#!/usr/bin/env python
from subprocess import check_call

cmd = "sox input.wav - -t wav pad 0.0 3.0 | sox - -t wav out.wav trim 0.0 3.0"
check_call(cmd, shell=True)

If you can't make it to write/read to/from stdout/stdin then you could try a named pipe, to avoid waiting for a file:

#!/bin/sh
mkfifo /tmp/sox.fifo
sox /tmp/sox.fifo -t wav out.wav trim 0.0 3.0 & # read from fifo
sox input.wav /tmp/sox.fifo pad 0.0 3.0         # write to fifo

or the same in Python:

#!/usr/bin/env python
from subprocess import Popen, check_call

with named_pipe() as path:
    trim = Popen(["sox", path, '-t', 'wav', 'out.wav', 'trim', '0.0', '3.0'])
    check_call(["sox", "input.wav", path, "pad", "0.0", "3.0"]) # pad
rc = trim.wait()

where named_pipe() is defined as:

import os
from contextlib import contextmanager
from shutil     import rmtree
from tempfile   import mkdtemp

@contextmanager
def named_pipe():
    dirname = mkdtemp()
    try:
        path = os.path.join(dirname, 'named_pipe')
        os.mkfifo(path)
        yield path
    finally:
        rmtree(dirname)

score 1 · Answer 2 · answered Mar 15 '14 at 14:08

1

You can wait file before it will be avaiable for reading:

handle = open(tmp_file.name)
from subprocess import select
select.select([handle], [], [])

answered Mar 15 '14 at 14:08

mingaleg

2,017
16
28

1

Can you elaborate a little bit? I'm not sure how I can know when the file will be available for reading? Does `select.selec([handle],[],[])` return something? – Sennster Mar 15 '14 at 21:15
1

do not import `select` from `subprocess` module, use `import select` instead. – jfs Mar 16 '14 at 09:19
1

if `f = open(filename)` succeeds then `select.select([f], [], [])` may return immediately i.e., you can't use it to wait for a file. – jfs Mar 16 '14 at 09:23

Wait for external program to finish writing to file

2 Answers2

Linked