How to get subprocess stdout while running git command?

Question

I have a program written in python and used git command in it.. For some reason I don't want to use git-python or others instead of subprocess. But I'm currently stuck in getting git clone output.

I've tried some code snippet. Some works fine with commands like ping 8.8.8.8, but not the git clone.

for example

using thread

def log_worker(stdout):
    while True:
        last = non_block_read(stdout).strip() 
        if last != "":
            print(last)


def non_block_read(output):
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    try:
        return output.read()
    except:
        return ''

def test():
    mysql_process = subprocess.Popen(
        "ping google.com",
        shell=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)

    thread = Thread(target=log_worker, args=[mysql_process.stdout])
    thread.daemon = True
    thread.start()

    mysql_process.wait()
    thread.join(timeout=1)

test()

or

newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            print('tt')
            out = []
            last = stream.read(1)
            # Don't loop forever
            if last == '' and proc.poll() is not None:
                break
            print('last', last)
            while last not in newlines:
                print("loop")
                # Don't loop forever
                if last == '' and proc.poll() is not None:
                    break
                out.append(last)
                last = stream.read(1)
            out = ''.join(out)
            yield out

def example():
    cmd = ['ls', '-l', '/']
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        # Make all end-of-lines '\n'
        universal_newlines=True,
        shell = True
    )
    for line in unbuffered(proc):
        print('new line')
        print line

example()

and most common one

for line in iter(proc.stdout.readline, ''):
    sys.stdout.write('{:.2f}  {}\n'.format(
        time.time() - start,
        line.rstrip()
    ))
    sys.stdout.flush()

all of them works fine with ping google.com, but not git clone. Is there any way to solve this? Thanks in advance!

UPDATE1: In face, I'm just want to get the finished percent of git clone. Log or any log files are not needed.

Answer 1

When not writing to a terminal, git clone doesn't have any output to either stdout or stderr, except on error.

When writing to a terminal, of course, it has lots of output—but that output is progress bars that are continually overwritten. Usually, you don't want that—it's going to be a big mess of control characters and repeated lines.

But if you do want it, there are two options.

---

First, you can use a PTY (Pseudo-TTY). You can create a PTY with os.openpty, then hand the PTY off explicitly to the child process. Or you can use os.forkpty, which handles forking and automatically hooking up the PTY so all you have to do is call one of the os.exec functions. Or you can use the pty module. (It's not entirely clear which is more portable; openpty and forkpty claim that pty is more portable, and conceptually it's designed that way… but it's also only really tested on Linux.)

Note that git wants the PTY as its stderr, not its stdout.

---

Alternatively, most git commands have a --progress flag that causes them to write progress to stderr even if it's not a terminal. At least as of the version documented here, this includes clone, but of course you should check the man for your local version. So, that may be all you need. (Also see the --verbose flag.)

However, this may not be as nice. For me, when I provide a PTY with no attached termcaps, I get each line followed by a \r without \n to overwrite it; when I use the --progress option, git detects the termcaps of whatever terminal my script happens to be running it, which means I end up getting ANSI color codes as well as \rs.

---

Of course either way I'm getting hundreds of useless lines that I have no interest in, but I assume that's what you wanted? (Or maybe you want to use universal_newlines='\r' to translate the '\r' to '\n'? That's slightly cheating, because this is self-overwriting Unix terminal output, and you're pretending it's classic-Mac output… but it works.)

Answer 2

I had a similar issue, but I also wanted real-time progress because the git clone was taking a long time (like 5 minutes), since it was a large git repo. And I wanted to provide real-time feedback to the user.

Here is a Python 3.7 working example:

# This will print stdout/stderr as it comes in
def run_shell_command_with_realtime_output(shell_command_string, working_dir='.'):
    # print the command to be executed, just for fun
    print("run_shell_command >", shell_command_string)

    # run it, this will NOT block
    sub_process = subprocess.Popen(shell_command_string,
                                   shell=True,
                                   cwd=working_dir, universal_newlines=True,
                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    # print the stdout/stderr as it comes in
    while True:
        # The readline() will block until...
        # it reads and returns a string that ends in a '\n',
        # or until the process has ended which will result in '' string
        output = sub_process.stdout.readline()
        if output:
            print(output.strip())
        elif sub_process.poll() is not None:
            break

    # get the return code
    return_code = sub_process.wait()

    # Was there an error?
    if return_code != 0:
        print("FYI, the subprocess had an error, you might want to do something special...")

    # return the sub_process, in case the caller wants to check exit/return codes
    return sub_process

It turns out git clone does not seem to write to stdout/stderr like we normally are used to. Instead it uses a pager, which is how it updates the same line, like when it is cloning, and the % is incrementing on the same line.

So you need to call it like this.

# prepare to clone the git repo
git_clone_url_with_credentials = "https://<username>:<password>@bitbucket.org/myreponame.git"
git_clone_with_progress_cmd = "git --no-pager clone --progress {}".format(git_clone_url_with_credentials)
helpers_misc.run_shell_command_with_progress(git_clone_with_progress_cmd)

And that should do it...