Why doesn't `stdin.read()` read entire buffer?

Question

I've got the following code:

def get_input(self):
    """
    Reads command from stdin, returns its JSON form
    """
    json_string = sys.stdin.read()
    print("json string is: "+json_string)
    json_data =json.loads(json_string)
    return json_data

def accept_commands(self):
    while True:
        json_data = self.get_input()
        command = self.command_analyzer.is_command(json_data) # Check wether the command exists. Return it if it does
        #The command exists
        if command is not None:
            #The addon is not currently active
            if analyzer.intent(json_data) not in self.active_addons:
                self.activate_addon(command,json_data)

            #The addon is active and so we need to send the data to the subprocess
            else:
                self.communicate_with_addon(command,json_data,json_string)

It reads a json string that was sent to it from another process. The json is read from stdin. For some reason I get the following output:

json string is: <Some json here>
json string is: 
Traceback (most recent call last):
  File "/Users/Matan/Documents/workspace/ProjectSH/addonmanager/addon_manager.py", line 63, in <module>
    manager.accept_commands()
  File "/Users/Matan/Documents/workspace/ProjectSH/addonmanager/addon_manager.py", line 49, in accept_commands
    json_data = self.get_input()
  File "/Users/Matan/Documents/workspace/ProjectSH/addonmanager/addon_manager.py", line 42, in get_input
    json_data =json.loads(json_string)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")

The json is sent from the following:

class MessageReceiver:

    def __init__(self):
        '''
        Connect to the AMQP broker and starts listening for messages.
        Creates the a Popen object to pass command info to the addon_manager script (which
        is in charge of managing scripts)
        '''
        addon_manager_path = configuration.addon_manager_path()
        addon_manager_path = os.path.join(addon_manager_path,'addon_manager.py')
        execute = "python " + addon_manager_path
        self.addon_manager = subprocess.Popen(execute, stdin=subprocess.PIPE, shell=True)


        self.component_name= configuration.get_attribute("name")

        if len(sys.argv)>1:
            host_ip = sys.argv[1]
        else:
            host_ip = 'localhost'

        #Start a connection to the AMQP server
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=host_ip))

        #Create a channel to the server
        self.channel = self.connection.channel()

        self.channel.queue_declare(queue="kitchen")

        #callback method to be called when data is received
        #It sends the data that is received by the client to the addon_manager
        def data_received(ch, method, properties, body):
            ##TODO: Might want to add some checks. Is body a JSON? etc.
            print("writing data to addon_manager")
            self.addon_manager.communicate(body)


        self.channel.basic_consume(data_received,queue='kitchen',no_ack=True)


        self.channel.start_consuming()

What's wrong here?

Answer 1

By default, stdin.read() blocks until the entire buffer has been read. If all you could decode was one JSON object, then that is all that was sent before stdin was closed by the other process.

If you need to stream multiple JSON blocks, you should

a) not close the stream from the writing process, and b) not do a blocking read in Python.

Read in chunks or lines instead; see Loading and parsing a JSON file with multiple JSON objects in Python and How do I use the 'json' module to read in one JSON object at a time? for techniques for either reading JSON objects that are line-delimited or where multiple lines are used.

You can adapt either to produce a generator function; you'd loop over that to yield one JSON object at a time, blocking in between to wait for the next JSON object:

def get_input(self):
    for line in sys.stdin:
        yield json.loads(line)

and

def accept_commands(self):
    for json_data in self.get_input():
        # do something with `json_data`

You are using Popen.communicate() to write to the pipe. This closes the pipe after writing to it, after which it waits for the process to terminate.

If you expected the pipe to remain open, don't use Popen.communicate and instead write directly to the Popen.stdin pipe.

Answer 2

It seems like you are calling get_input twice and the second there is nothing in the buffer. You should check if there is something before you try to parse JSON:

def get_input(self):
    """
    Reads command from stdin, returns its JSON form
    """
    json_string = sys.stdin.read()
    if json_string:
        print("json string is: "+json_string)
        json_data =json.loads(json_string)
        return json_data
    else:
        return None