rsync called by subprocess Popen works when running script but does not when I generate an app with py2app

Question

This is my code:

def uploadByRSync(host, user, passwd, src, dst, statusManager):
    try:
        os.environ["RSYNC_PASSWORD"] = passwd
        print host, user, passwd, src, dst
        parameters = ["rsync", "-azP", "--partial", src ,"{3}@{0}::{2}/{1}".format(host, dst, user, user)]
        print " ".join(parameters)
        process = subprocess.Popen(parameters, stdout=subprocess.PIPE,  stderr=subprocess.STDOUT,  universal_newlines=True)
        for line in unbuffered(process):
            if "%" in line:
                spl = line.split()
                statusManager.uploadSpeed = spl[2]
                statusManager.uploaded = spl[1]
        return not process.wait()
    except Exception as ex:
        print ex
        return False 


newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            out = []
            last = stream.read(1)
            # Don't loop forever
            if last == '' and proc.poll() is not None:
                break
            while last not in newlines:
                # Don't loop forever
                if last == '' and proc.poll() is not None:
                    break
                out.append(last)
                last = stream.read(1)
            out = ''.join(out)
            print out
            yield out

When running with the py2app version I can never get an output. When running as script everything works just fine. ps: this code runs on a separated thread of a Qt app. Does anyone have any idea why this is happening?

did you check whether py2app uses the same python version as when you run the interpreter? Checking `last == ''` might behave differently.. — frans, Feb 09 '15 at 10:56

score 0 · Answer 1 · edited Feb 09 '15 at 18:55

0

Just made a test changing the Popen call by a simple 'ls',but I still cannot get the output when running py2app version. It works just fine when running python script. When I kill the py2app version app the output is just printed.

process = subprocess.Popen(["ls"], stdout=subprocess.PIPE,  stderr=subprocess.STDOUT,  universal_newlines=True)

edited Feb 09 '15 at 18:55

kguest

3,804
3
29
31

answered Feb 09 '15 at 10:58

Igor Zanotto

1

1

Please use comments or edit your post to add additional information rather than posting an answer which is not an answer. – frans Feb 09 '15 at 11:14
please, [edit your question](http://stackoverflow.com/posts/28407988/edit) and include this info there. Then delete this non-answer. – jfs Feb 09 '15 at 15:43

frans · Answer 2 · 2015-02-09T11:16:43.670

Most likely you have a stream buffering issue. Here is how you can output all lines of your process in real time:

    import subprocess
    import select
    p = subprocess.Popen(parameters, 
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE, 
                         bufsize=0)

    poll = [p.stdout.fileno(), p.stderr.fileno()]

    while True:
        # check if process is still running and read remaining data
        if p.poll() is not None: 
            for l in p.stdout.readlines():
                print(l)
            for l in p.stderr.readlines():
                print(l)
            break

        # blocks until data is being recieved
        ret = select.select(poll, [], [])

        for fd in ret[0]:
            line = p.stdout.readline() if fd == p.stdout.fileno() else p.stderr.readline()
            print(line)

The code in the question already reads one byte at a time. `select` won't force the child process to flush its standard streams buffers any faster i.e., if OP's code fails; your code may also fail. `bufsize=0` by default on Python 2. I don't know what effect `universal_newlines=True` has on the buffering. OP uses `stderr=subprocess.STDOUT` therefore a [simpler code could be used](http://stackoverflow.com/a/17698359/4279) — jfs, Feb 09 '15 at 21:11

rsync called by subprocess Popen works when running script but does not when I generate an app with py2app

2 Answers2