Why does writing to a file work, but writing to an HTTP server request handler not work?

Question

I'm using a Raspberry Pi Picamera board to capture data, using the following code:

with picamera.PiCamera(
    sensor_mode=4,
    resolution='1640x1232',
    framerate=30
    ) as camera:

    camera.rotation = 180
    camera.start_recording(StreamingOutput(), format='mjpeg')

try:
    server = StreamingServer(('', 8000), StreamingHandler)
    server.serve_forever()
finally:
    camera.stop_recording()


class StreamingOutput:
    def __init__(self):
        self.frame = None
        self.condition = threading.Condition()
        self._buffer = io.BytesIO()

    def write(self, buf):
        if buf.startswith(b'\xff\xd8'):
            # New frame, copy the existing buffer's content and notify all
            # clients it's available
            self._buffer.truncate()
            with self.condition:
                self.frame = self._buffer.getvalue()
                self.condition.notify_all()
            self._buffer.seek(0)


class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer):
    allow_reuse_address = True
    daemon_threads = True


class StreamingHandler(server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/capture.jpg':
            with self._output.condition:
                self._output.condition.wait()
                frame = self._output.frame

            # This works.
            with open("frame.jpg", 'wb') as f:
                f.write(frame)

            # This produces a truncated image.
            self.send_response(200)
            self.send_header('Content-Type', 'image/jpeg')
            self.send_header('Content-Length', len(frame))
            self.end_headers()
            self.wfile.write(frame)

It's the damndest thing: although the image will save to disk just fine (frame.jpg is completely fine), it'll produce a truncated image if going through the HTTP server. Here's a screenshot:

I've tried a number of different things, and I'm at a dead end. Any ideas?

The code you posted can't possibly be writing the frame to disk "just fine", because you're not opening the file in write mode. — jasonharper, Jan 23 '20 at 00:16

score 1 · Accepted Answer · answered Jan 23 '20 at 00:43

1

My guess is that you are running into a memory issue. When you write to disk the content is streamed, but I expect that when using wfile.write(frame) you are having to nearly double the memory used since you aren't chunking the data (instead of having frame and a chunk of frame in memory at any given time you have two copies. I would try using shutil to see if this corrects the issue by performing shutil.copyfileobj(frame, self.wfile). This is just a guess but hopefully it fixes your problem! shutil docs

answered Jan 23 '20 at 00:43

jteezy14

436
4
11

Interesting idea. `frame` isn't a file-like object, so just throwing in `copyfileobj` causes `AttributeError: 'bytes' object has no attribute 'read'`. I'll try to see if I can convert frame into something else to test your hypothesis of memory issues. – clam Jan 23 '20 at 00:53
Just to test it you could write it to disk (like you are already) and then use that copy on disk with shutil. I wouldn't make that the final solution but it may be the easiest way to test it. – jteezy14 Jan 23 '20 at 00:57
That did work; the JPEG is now successfully streaming through the HTTP server. Now all that's left to do is figure out why; constantly writing the file to disk is a surefire way to bork the SD card. Maybe I could wrap the frame object in a streamer of some kind. – clam Jan 23 '20 at 01:04
I'm glad that worked. I agree, that is definitely not the long term solution but a good test. You probably need something like an in memory file. Here is a link to another post that might get you on the right track https://stackoverflow.com/questions/44672524/how-to-create-in-memory-file-object/44672691 – jteezy14 Jan 23 '20 at 15:29

score 1 · Answer 2 · answered Jan 23 '20 at 01:24

Here's the working version:

def do_capture(self):
    with self._output.condition:
        self._output.condition.wait()
        frame = self._output.frame

    self.send_response(200)
    self.send_header('Content-Type', 'image/jpeg')
    self.send_header('Content-Length', len(frame))
    self.end_headers()
    fb = io.BytesIO(frame)
    shutil.copyfileobj(fb, self.wfile)

Wrapping frame in a BytesIO stream works in conjunction with shutil, which expects two file-like objects. Together, it produces a functional stream.

Why does writing to a file work, but writing to an HTTP server request handler not work?

2 Answers2