1

I want to use ion-js but it a sync library over strings or ArrayBuffers, not a Stream or ReadableStream.

I thought to use worker_threads, SharedArrayBuffer, and Atomics to create a ArrayBuffer that is filled in a background thread and blocks waiting for bytes.

class BlockingArrayBuffer implements ArrayBuffer {
  constructor(
    private readonly dataBuffer: Uint8Array,
    private readonly availableBytes: Int32Array,
  ) {  }

  get byteLength(): number {
    return this.dataBuffer.byteLength;
  }

  slice(begin: number, end: number = this.byteLength): ArrayBuffer {
    // calls Atomics.wait till availableBytes is greater than begin and end.
    this.blockTillAvailable(Math.max(begin, end));
    return this.dataBuffer.slice(begin, end);
  }
}

There's a worker thread not shown that reads the stream and updates the two shared array buffers backing dataBuffer and availableBytes.

This actually works when i call slice directly.

const {dataBuffer, availableBytes} = await makeWorker();
const blocking = new BlockingArrayBuffer(dataBuffer, availableBytes);
blocking.slice(0, 10); // completes quickly.
blocking.slice(0);     // blocks till buffer is full.

However it doesn't work when I wrap the blocking array with a typed array

console.log(blocking.slice(0)); // prints  filled array
console.log(new Uint8Array(blocking).slice(0)); // Uint8Array(0) []
everett1992
  • 2,351
  • 3
  • 27
  • 38
  • 1
    As a side note, have you given the usability of this solution consideration? Unless I misunderstand what you mean, you plan to block the main thread, which would in turn freeze the whole page UI for the user until this operation finishes. – Etheryte Feb 10 '21 at 22:05
  • 1
    ion is a binary format, ion-js has methods to read one field at a time, so we can block till the first 100 byte field is available, read and process a field, then block till the next 100 byte field is available. That may lock up the UI more than awaiting a promise, but I'm writing a node CLI and the main thread needs to process the fields in order. – everett1992 Feb 10 '21 at 22:14
  • I really don't know this ion thing nor what it's supposed to do, but are you sure it will read the data linearly from start to end? If it needs to jump inside the data all you are doing sounds kinda moot since it would require the full data to be available completely before it can process it anyway. – Kaiido Feb 11 '21 at 00:44
  • Say I want to process a ArrayBuffer in 10 byte chunks: `for (let i = 0; i < buffer.byteLength; i += 10) { await process(buffer.slice(i, i + 10)) }` With a normal array buffer you have to wait for all bytes to be available before processing the very first chunk. But with my blocking array buffer the main thread will wait till 10 bytes are available in the buffer, and process the first chunk. While the main thread is processing the first chunk the worker is adding more bytes to the buffer – everett1992 Feb 11 '21 at 01:18
  • `implements ArrayBuffer` is not the same as `extends ArrayBuffer`. I'm surprised `new Uint8Array(blocking)` works at all. – Bergi Feb 11 '21 at 03:32
  • @Bergi I have tried `extends ArrayBufer` with the same behavior. – everett1992 Feb 11 '21 at 04:33

0 Answers0