Convert ArrayBuffer into ImageData for drawing on canvas: optimization

Question

I am streaming video over a WebSocket by sending each frame in the raw ImageData format (4 bytes per pixel in RGBA order). When I receive each frame on the client (as an ArrayBuffer), I want to paint this image directly onto the canvas as efficiently as possible, using putImageData.

This is my current solution:

// buffer is an ArrayBuffer representing a properly-formatted image
var array = new Uint8ClampedArray(buffer);
var image = new ImageData(array, width, height);
canvas.putImageData(image, 0, 0);

But it is rather slow. My theories as to why:

• the array (which is ~1MB in size) is being copied thrice, once into the Uint8ClampedArray, once into the ImageData, and lastly into the canvas, each frame (30 times per second).

• I am using new twice for each frame, which may be a problem for the garbage collector.

Are these theories correct and if so, what tricks can I employ to make this as fast as possible? I am willing to accept an answer that is browser-specific.

Answer 1

No, both your ImageData image and your TypedArray array share the exact same buffer buffer.

These are just pointers, your original buffer is never "copied".

var ctx = document.createElement('canvas').getContext('2d');

var buffer = ctx.getImageData(0,0,ctx.canvas.width, ctx.canvas.height).data.buffer;

var array = new Uint8ClampedArray(buffer);

var image = new ImageData(array, ctx.canvas.width, ctx.canvas.height);

console.log(array.buffer === buffer && image.data.buffer === buffer);

For your processing time issue, the best way would be to simply send directly the video stream to a videoElement and use drawImage.