Is it normal for an HTMLCanvasElement captured stream to lag on Chrome when the Canvas is hidden and drawn onto by its OffscreenCanvas in a web worker

Question

Ok so the question is a bit much, but it actually mentions every part of the "bug" I'm experiencing. I'm not sure if it is really a bug or if there is something I don't understand yet. I created a demo on codepen to reproduce the weird behavior.

https://codepen.io/rroyerrivard/pen/jOwBLbB

Like I wrote in the codepen, there seems to be a bug in Chrome for which the stream of an HTMLCanvasElement does not get refreshed on every image draw when it is hidden. But for that to happen, there needs to be these 4 conditions all at once.

• We must feed an HTMLVideoElement with a MediaStream gotten from a call to HTMLCanvasElement.captureStream() (instead of directly showing the HTMLCanvasElement).
• The HTMLCanvasElement from which we get the MediaStream must be hidden (either not in the DOM or having it hidden with css).
• We must draw on the HTMLCanvasElement from an OffscreenCanvas that we get from a call to HTMLCanvasElement.transferControlToOffscreen().
• The draw on the OffscreenCanvas must be done in a web worker that got the OffscreenCanvas transferred to.

I was unfortunate enough to hit all these conditions at once in a web app at work. I can avoid the bug by not using the transferControlToOffscreen() call and draw an ImageBitmap in the main thread after receiving it from the web worker, but that reduces the FPS by roughly 18%.

Is this a known bug? Is there a way to force the MediaStream to refresh on every draw of the OffscreenCanvas?

Answer 1

I guess it is expected behavior yes.

The thing here is that you made your worker thread wait using setTimeout and MessageEvents from the main thread.
The OffscreenCanvas will commit its bitmap to the placeholder canvas in the Worker's rendering frame. But by default the Worker won't enter this rendering frame. You need to request it, by using requestAnimationFrame.
Having the placeholder visible in the page will internally make the request for the OffscreenCanvas to commit its bitmap when the placeholder canvas will itself get rendered (i.e in the main thread's rendering frame), that's why it works when the placeholder canvas is visible.

Note that we used to have a OffscreenCanvas.commit() method but it has been deprecated when requestAnimationFrame made its way in WorkerContexts.

So use requestAnimationFrame in your Worker to actually force the commit of the bitmap to the placeholder canvas.

const video = document.querySelector("video");
const select = document.querySelector("select");
const canvas = document.createElement("canvas");
const offscreen = canvas.transferControlToOffscreen();
const worker = new Worker(getWorkerURL());
worker.postMessage({ offscreen }, [offscreen]);
select.oninput = e => worker.postMessage({ method: select.value });
worker.onmessage = (evt) => {
  video.srcObject = canvas.captureStream();
};

function getWorkerURL() {
  return URL.createObjectURL(
    new Blob([
      document.querySelector("[type='text/worker']").textContent
    ], { type: "text/javascript" })
  );
}

canvas { border: 1px solid }

<video controls autoplay></video>
<label>waiting method:<select><option>rAF</option><option>setTimeout</option></select></label>
<script type="text/worker">
  let ctx;
  let waiting_method = "rAF";
  onmessage = ({ data: { offscreen, method } }) => {
    if (offscreen) {
      ctx = offscreen.getContext("2d");
      draw();
      postMessage("ready");
    }
    else if (method) {
      waiting_method = method;
    }
  };
  function draw() {
    ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);
    ctx.fillText(new Date().getTime(), 30, 30);
    if (waiting_method === "rAF") {
      requestAnimationFrame(draw);
    }
    else {
      setTimeout(draw, 1000/30);
    }
  }
</script>

---

Now, I guess we could also expect the call to captureStream() to actually trigger the same internal request for commit that the visible placeholder canvas triggers, so you may want to file an issue at https://crbug.com regardless.