2

I'm trying to run a few ffmpeg commands in parallel, using Cygwin and Python 2.7.

This is roughly what I have:

import subprocess
processes = set()

commands = ["ffmpeg -i input.mp4 output.avi", "ffmpeg -i input2.mp4 output2.avi"] 

for cmd in commands:
  processes.add(
    subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
  )

for process in processes:
  if process.poll() is None:
    process.wait()

Now, once I am at the end of this code, the whole program waits. All the ffmpeg processes are created, but they're idle, i.e., using 0% CPU. And the Python program just keeps waiting. Only when I hit Ctrl-C, it suddenly starts encoding.

What am I doing wrong? Do I have to "send" something to the processes to start them?

jfs
  • 399,953
  • 195
  • 994
  • 1,670
slhck
  • 36,575
  • 28
  • 148
  • 201
  • unrelated: 1- don't use `shell=True`, pass a list argument instead (`shlex.split(cmd)` should work in your case) 2- do not use `stdout=PIPE` unless you *read* from the pipe ([to discard the output, use `stdout=DEVNULL` instead](http://stackoverflow.com/q/11269575/4279)) 3- drop `process.poll()` -- it is unnecessary here. 4- you might want to add `stdin=DEVNULL` 5- you might want to [limit the number of concurrent `ffmpeg` processes](http://stackoverflow.com/q/14533458/4279) – jfs Feb 04 '16 at 14:09
  • Thanks for all these pointers. But why should I drop `process.poll()`? What should I do instead to wait for all of the processes to finish? – slhck Feb 04 '16 at 14:58
  • `process.wait()` is enough. [Click the last link in my previous comment, to see the complete code example](http://stackoverflow.com/q/14533458/4279) – jfs Feb 04 '16 at 15:00

2 Answers2

6

This is only a guess, but ffmpeg usually produces a lot of status messages and output on stderr or stdout. You're using subprocess.PIPE to redirect stdout and stderr to a pipe, but you never read from those, so if the pipe buffer is full, the ffmpeg process will block when trying to write data to it.

When you kill the parent process the pipe is closed on its end, and probably (i haven't checked) ffmpeg handles the error by just not writing to the pipe anymore and is therefore unblocked and starts working.

So eiter consume the process.stdout and process.stderr pipes in your parent process, or redirect the output to os.devnull if you don't care about it.

Community
  • 1
  • 1
mata
  • 67,110
  • 10
  • 163
  • 162
  • What I did was remove the pipe redirection altogether. This fixed it. Not sure where I got the initial code from … Thanks! – slhck Feb 04 '16 at 14:58
  • I think your diagnosis (buffer overflow due to ffmpeg's weird output) is right, sir. – yPhil Jun 24 '16 at 13:15
3

In addition to what @mata says, ffmpeg may also be asking you if you want to overwrite output.avi and waiting on you to type "y". To force-overwrite, use the "-y" command-line option (ffmpeg -i $input -y $output).

Community
  • 1
  • 1
Ronald S. Bultje
  • 10,828
  • 26
  • 47