45

Today I visited jsPerf and now I am wondering…

  1. What is "ops/sec"?
  2. How many iterations does it do?
  3. On what basis does it calculate which is faster? What is the formula behind these calculations?

Example: http://jsperf.com/concatenation-vs-join

Can anyone tell me?

ggorlen
  • 44,755
  • 7
  • 76
  • 106
Mohit Kumar
  • 1,885
  • 5
  • 21
  • 24

2 Answers2

129

I wrote Benchmark.js, which jsPerf uses.

  1. "ops/sec" stands for operations per second. That is how many times a test is projected to execute in a second.

  2. A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5 seconds (configurable), or at least 5 runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000 times in 50 ms (the minimum run time for most environments), and then repeated 100 times more (5 seconds). A larger sample size (in this example, 100), leads to a smaller margin of error.

  3. We base the decision of which test is faster on more than just ops/sec by also accounting for margin of error. For example, a test with a lower ops/sec but higher margin of error may be statistically indistinguishable from a test with higher ops/sec and lower margin of error.

    We used a welch t-test, similar to what SunSpider uses, but switched to an unpaired 2-sample t-test for equal variance (the variance is extremely small) because the welch t-test had problems comparing lower ops/sec and higher ops/sec with small variances which caused the degrees of freedom to be computed as less than 1. We also add a 5.5% allowance on tests with similar ops/sec because real world testing showed that identical tests can swing ~5% from test to re-test. T-tests are used to check that differences between tests are statistically significant.

Community
  • 1
  • 1
John-David Dalton
  • 23,993
  • 3
  • 23
  • 13
  • 1
    I've always wondered if when doing tests that manipulate i.e. HTML form elements, do you need to "reset" them back to defaults in order for the other tests to be fair/accurate? – Gary Green Jun 03 '11 at 12:31
  • 1
    @GaryHole Yes, you would need to reset everything back to the default state outside of the timed code region for it to be as accurate as possible. – Mathias Bynens May 29 '13 at 08:15
  • What is the ±percentage inside the Ops/sec column? – BornToCode Dec 02 '15 at 16:02
6

You can read Bulletproof JavaScript benchmarks article from the authors. It uses Benchmark.js btw, which is Open Source.

Mathias Bynens
  • 144,855
  • 52
  • 216
  • 248
gblazex
  • 49,155
  • 12
  • 98
  • 91