Setting stdout to non-blocking in python

Question

Prior warning: I'm hacking around here out of curiosity. I have no specific reason to do what I'm doing below!

Below is done on Python 2.7.13 on MacOS 10.12.5

I was hacking around with python and I thought it'd be interesting to see what happened if I made stdout nonblocking

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

The call to fcntl is definitely successful. I then try to write a large amount of data (bigger than the max buffer size of a pipe on OSX - which is 65536 bytes). I do it in a variety of ways and get different outcomes, sometimes an exception, sometimes what seems to be a hard fail.

Case 1

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
    sys.stdout.write("A" * 65537)
except Exception as e:
    time.sleep(1)
    print "Caught: {}".format(e)

# Safety sleep to prevent quick exit
time.sleep(1)

This always throws the exception Caught: [Errno 35] Resource temporarily unavailable. Makes sense I think. Higher level file object wrapper is telling me the write call failed.

Case 2

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
    sys.stdout.write("A" * 65537)
except Exception as e:
    print "Caught: {}".format(e)

# Safety sleep to prevent quick exit
time.sleep(1)

This sometimes throws the exception Caught: [Errno 35] Resource temporarily unavailable or sometimes there is no exception caught and I see the following output:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

Case 3

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
    sys.stdout.write("A" * 65537)
except Exception as e:
    print "Caught: {}".format(e)

# Safety sleep to prevent quick exit
time.sleep(1)

print "Slept"

This sometimes throws the exception Caught: [Errno 35] Resource temporarily unavailable or sometimes there is no exception caught and I just see "Slept". It seems that by printing "Slept" I don't get the error message from Case 2.

Case 4

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
    os.write(sys.stdout.fileno(), "A" * 65537)
except Exception as e:
    print "Caught: {}".format(e)

# Safety sleep to prevent quick exit
time.sleep(1)

Always okay!

Case 5

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
    print os.write(sys.stdout.fileno(), "A" * 65537)
except Exception as e:
    print "Caught: {}".format(e)

# Safety sleep to prevent quick exit
time.sleep(1)

This is sometimes okay or sometimes prints the close failed in file object destructor error message.

My question is, why does this fail like this in python? Am I doing something fundamentally bad here - either with python or at the system level?

It seems like somehow that writing too soon to stdout when the write already failed causes the error message. The error doesn't appear to be an exception. No idea where it's coming from.

N.B. I can write the equivalent program in C and it works okay:

#include <stdio.h>
#include <stdlib.h>
#include <memory.h>
#include <sys/fcntl.h>
#include <unistd.h>

int main(int argc, const char * argv[])
{
    const size_t NUM_CHARS = 65537;
    char buf[NUM_CHARS];

    // Set stdout non-blocking
    fcntl(fileno(stdout), F_SETFL, O_NONBLOCK);

    // Try to write a large amount of data
    memset(buf, 65, NUM_CHARS);
    size_t written = fwrite(buf, 1, NUM_CHARS, stdout);

    // Wait briefly to give stdout a chance to be read from
    usleep(1000);

    // This will be written correctly
    sprintf(buf, "\nI wrote %zd bytes\n", written);
    fwrite(buf, 1, strlen(buf), stdout);
    return 0;
}

Answer 1

This is interesting. There are a few things that I've found so far:

Case 1

This is because sys.stdout.write will either write all of the string or throw an exception, which isn't the desired behavior when using O_NONBLOCK. When the underlying call to write returns EAGAIN (Errno 35 on OS X), it should be tried again with the data that is remaining. os.write should be used instead, and the return value should be checked to make sure all the data is written.

This code works as expected:

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

def stdout_write(s):
    written = 0
    while written < len(s):
        try:
            written = written + os.write(sys.stdout.fileno(), s[written:])
        except OSError as e:
            pass

stdout_write("A" * 65537)

Case 2

I suspect that this error message is due to https://bugs.python.org/issue11380:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

I'm not sure why it's sometimes being called. It may be because there is a print in the except statement which is trying to use the same stdout that a write just failed on.

Case 3

This is similar to Case 1. This code always works for me:

fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

def stdout_write(s):
    written = 0
    while written < len(s):
        try:
            written = written + os.write(sys.stdout.fileno(), s[written:])
        except OSError as e:
            pass

stdout_write("A" * 65537)

time.sleep(1)

print "Slept"

Case 4

Make sure you check the return value of os.write, I suspect that the full 65537 bytes are not being successfully written.

Case 5

This is similar to Case 2.