Java Microbenchmarking Harness vs System.getNanotime()

Question

Question1: Why JMH better than simple System.getNanotime()?

Question2: What can I conclude from the results (look at benchmarking results section) besides validateLongKeyBinary is 64 percents faster than validateLongKeyAscii?

Example (Code):

import net.spy.memcached.util.StringUtils;
import org.openjdk.jmh.annotations.GenerateMicroBenchmark;

public class KeyBench {

private static final String LONG_KEY = "thisIsAFunkyKeyWith_underscores_AndAlso334" +
            "3252545345NumberslthisIsAFunkyKeyWith_underscores_AndAlso3343252545345Numbe" +
            "rslthisIsAFunkyKeyWith_underscores_AndAlso3343252545345NumberslthisIsAFunkyK" +
            "eyWith_underscores_AndAlso3343252545345Numbersl";


    @GenerateMicroBenchmark
    public void validateLongKeyBinary() {
        StringUtils.validateKey(LONG_KEY, true);
    }

    @GenerateMicroBenchmark
    public void validateLongKeyAscii() {
        StringUtils.validateKey(LONG_KEY, false);
    }
}

Benchmarking results

# Running: benchmarks.KeyBench.validateLongKeyAscii

Result : 393,667 ±(95%) 13,985 ±(99%) 20,094 ops/ms
  Statistics: (min, avg, max) = (357,445, 393,667, 413,004), stdev = 19,552
  Confidence intervals: 95% [379,682, 407,653], 99% [373,573, 413,762]


# Running: benchmarks.KeyBench.validateLongKeyBinary

Result : 644,023 ±(95%) 6,881 ±(99%) 9,887 ops/ms
  Statistics: (min, avg, max) = (621,784, 644,023, 654,178), stdev = 9,620
  Confidence intervals: 95% [637,142, 650,904], 99% [634,136, 653,910]

Benchmark                             Mode Thr     Count  Sec         Mean   Mean error    Units
b.KeyBench.validateLongKeyAscii      thrpt   1        10    1      393,667       20,094   ops/ms
b.KeyBench.validateLongKeyBinary     thrpt   1        10    1      644,023        9,887   ops/ms

Related: [How do I write a correct micro-benchmark in Java?](http://stackoverflow.com/q/504103/1065197). — Luiggi Mendoza, May 05 '14 at 19:29
You shouldn't make `LONG_KEY` static final as it could lead to optimisations. Just use a `private String`. 600k ops per ms = 0.6 ops / ns so each method call uses less than 5 cpu cycles (depending on your processor). That seems a bit too fast. — assylias, May 05 '14 at 19:30
Although someone who reads this question already knows it: It's JMH (not JHM), and about http://openjdk.java.net/projects/code-tools/jmh/ — Marco13, May 05 '14 at 20:19
@LuiggiMendoza thanks, but that reference is about how to write microbenchmarks well. I'm asking about why we should write microbenchmarks if we've laready had `System.getNanotime()`? — VB_, May 06 '14 at 07:19
@assylias thanks. But why we should use microbenchmarks over simple time measurement? — VB_, May 06 '14 at 07:20
The link provided by Luiggi explains all the pitfalls of micro benchmarking. Using a library helps avoiding those pitfalls (enough JIT warmup, putting measured stuff in a method, avoiding the effects of loop unrolling etc.). In the end those libraries use time measurement so you can do it manually - it's just more difficult to get right. — assylias, May 06 '14 at 07:57

Aleksey Shipilev · Accepted Answer · 2014-05-06T13:33:59.740

JMH maintainer here.

Let me ask the leading question: Why would one use the library, if you can code most of the things yourself? The answer is actually simple: of course you can write everything given the infinite time, but in practice we have to reuse code to fit into reasonable time.

Now, it would only seem that having two timestamps around the code is enough to measure its performance. However, you have to control what exactly you are measuring, e.g. whether you are still in the transitional warmup phase, does your code actually execute or you measure a hollow shell after the optimization, how statistically significant your effect is, etc. etc. etc. Good benchmark frameworks try to help with that.

You can have a glimpse of the issues you will have to face by looking at JMH Samples, our benchmarking talks, or the relevant SO answers. Oh, and using nanoTime is harder than you think.

Java Microbenchmarking Harness vs System.getNanotime()

1 Answers1