1

I am trying to benchmark an Object's member function using Benchmark.js. Testing the function is made difficult by several factors:

  • Creation of the object is asynchronous (I could mock that part)
  • The member function is expensive
  • The member function is smart enough to only run once

Let's say it looks like this:

class Something {

  constructor(){
    // async ops
    this.expensiveValue = null;
  }

  expensiveOperation () {

    if (this.expensiveValue === null) {
      // Do expensive operation
      this.expensiveValue = result; // a non-null value
    }

  }

}

Now, I want to benchmark expensiveOperation. But due to its limitations, I also need to "reset" the object each run.

As far as I can tell,benchmark doesn't support per-run setups. I feel like making the reset part of the run isn't the best practice either, because it pollutes what I'm actually trying to benchmark.

I've looked at Benchmark.setup, but that only executes per-cycle, not per-run.

Am I missing something? Is there another benchmark option I can use? Or am I approaching this incorrectly?

Inigo
  • 12,186
  • 5
  • 41
  • 70
TheJim01
  • 8,411
  • 1
  • 30
  • 54
  • I'm not familiar with benchmark.js but I think that to be able to accomplish what you're asking (resetting state that gets set by a static method like a singleton or memoization) would need to reset the entire runtime environment. If this is a node.js environment, that means forking child processes just to clear out the context. I don't know how you would accomplish this in a browser context. – zero298 Jan 29 '20 at 17:23

2 Answers2

2

I'm not going to accept this answer, because I don't have enough knowledge on the subject to be 100% positive, but I did want to share what I found. If this should be moved to my question or a comment, just ping me with a comment.

I think the reason this isn't possible is because of how Benchmark.js performs its timing.

From what I've read (both in text and in its code), Benchmark doesn't time and sum-up individual runs, but instead counts how many runs complete over a specified amount of time (default = 5 seconds). This avoids certain gotchas like low-precision timers/timestamps, run-time optimization, and floating point rounding errors.

So it can't simply subtract the time it takes to execute the per-run setup function, due to those reasons. It also can't pause its timer to allow the per-run setup to execute.

For these reasons, it seems that Benchmark.js does not support per-run setup functions, because doing so would throw too much of a wrench in its works and reduce the timing accuracy.

TheJim01
  • 8,411
  • 1
  • 30
  • 54
  • I can confirm that this is the case. In fact, [someone opened a Benchmark.js issue](https://github.com/bestiejs/benchmark.js/issues/183), and the maintainer said can't and closed the issue. – Inigo Apr 29 '20 at 05:00
2

You can't always get what you want... ♩

Benchmark.js does not perform setup/teardown per iteration of your function, but only for each benchmark.js cycle, which is usually whatever it can run in 5 seconds, which might be hundreds to thousands to millions of calls. It does this for very good reason, as explained by the authors of Benchmark.js in Bulletproof JavaScript benchmarks.

If you believe those reasons don't apply to you, then I'd ask why you are even using Benchmark.js. You could easily write a loop and measure the duration of each call and take the average yourself, in a handful of lines of code. You don't need a fancy library.

You can't always get what you want

And if you try sometime, you might find

You get ♫   what you need.   ♬♪

To be honest, yes, you are approaching this incorrectly. Your expensiveOperation () is designed to be efficient for all subsequent calls, so of course a good benchmark should reflect that. The cost of the first call is amortized over all subsequent calls. Benchmark.js will try to measure the efficiency of your method as designed. That's the point.

Think about your underlying goal, and why it is you want to reset for each iteration. You don't want to benchmark expensiveOperation (), but only this part of the method:

      // Do expensive operation
      this.expensiveValue = result; // a non-null value

So simply factor that out into a method or function, and benchmark that. :)

Inigo
  • 12,186
  • 5
  • 41
  • 70
  • You make a good point that I should just factor out the functionality and benchmark that. I was trying to use an existing object where that function lives, but I'm not benchmarking the object, I'm benchmarking its function. Now I just have to remember what I was doing that required this... :) – TheJim01 Apr 29 '20 at 15:29
  • glad to be of assistance. – Inigo Apr 29 '20 at 19:02
  • 1
    Just a note on your footnote that became a header: Creating a loop to measure execution time is hindered by the lack of high-resolution timers (see: [performance.now(), Reduced time precision](https://developer.mozilla.org/en-US/docs/Web/API/Performance/now)). This is the reason for approaching `Benchmark.js` from the beginning, and also why `Benchmark.js` works so well. The lack of precision in the timer is amortized over the cycle, so a rounding error of `0.1s` over `10000` executions becomes a difference of `0.000001s` against a single run of the function. – TheJim01 Apr 30 '20 at 13:35
  • 1
    True. I had a case where I needed to generate random input for the function being benchmarked, and didn't want that to be included in the measurement. I ended up generating 50k inputs in advance, and the closure I passed to Benchmark.js would simply pick the next input from the array (looping to the beginning each 50k). But in your case, given an "expensive" function, the high resolution wouldn't be significant. – Inigo Apr 30 '20 at 15:49
  • I moved footnotes to top as I realized afterward that they directly address the title question and ought not be footnotes. Given the nature of the site, it's important to make sense for future readers. – Inigo Apr 30 '20 at 15:49