Testing Java Reflection, Method Handles and Mirror performance

Question

I'm testing how much performance Method Handles (delivered with Java 7) have against Java Reflection and Mirror (A fluent API to work with reflection - http://projetos.vidageek.net/mirror/mirror/). So I don't know if I wrote the test code properly.

I use JHM tool to make this tests, and I shared the code and results in my gist: https://gist.github.com/garcia-jj/057dcab7f388e5cb42d1

I cached in setup method all lookups to increase performance.

The final time is almost equals between Reflection and Method Handles.

So my question is: my test is right? There is some problems to cache method lookup in my application? Or I need to lookup always when I need to use invokeExact? If I don't create the cache the performance is too low.

Thank you

@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 5, time = 1000, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 5, time = 1000, timeUnit = TimeUnit.MILLISECONDS)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
public class Reflecting {
    // testing method handle
    private MethodHandle mhConcat;
    private MethodHandle mhHashCode;

    // testing reflection     
    private Method rconcat;
    private Method rhashcode;

    // testing mirror api
    private MethodHandler mrConcat;
    private MethodHandler mrHashcode;

    @Setup
    public void setup() throws Exception {
        mhConcat = publicLookup().findVirtual(String.class, "concat", methodType(String.class, String.class));
        mhHashCode = publicLookup().findVirtual(Object.class, "hashCode", methodType(int.class));

        rconcat = String.class.getDeclaredMethod("concat", String.class);
        rhashcode = String.class.getDeclaredMethod("hashCode");

        mrConcat = new Mirror().on((Object) "x").invoke().method("concat");
        mrHashcode = new Mirror().on((Object) "xy").invoke().method("hashCode");
    }

    @GenerateMicroBenchmark
    public void invoke(BlackHole bh) throws Throwable {
        bh.consume((String) mhConcat.invokeExact("x", "y"));
        bh.consume((int) mhHashCode.invokeExact((Object) "xy"));
    }

    @GenerateMicroBenchmark
    public void reflect(BlackHole bh) throws Throwable {
        bh.consume(rconcat.invoke("x", "y"));
        bh.consume(rhashcode.invoke("xy"));
    }

    @GenerateMicroBenchmark
    public void mirror(BlackHole bh) throws Throwable {
        bh.consume(mrConcat.withArgs("y"));
        bh.consume(mrHashcode.withoutArgs());
    }
}

Answer 1

Well, the definition of "wrong" usually starts with the definition of what is "right". Assuming you want to measure the invocation costs for a target method via the different means, I can name a few things which may require attention:

1. @GenerateMicroBenchmark suggests you use quite outdated JMH (it was renamed to @Benchmark months ago). Consider updating to get the more reliable harness version.

2. The usage of Blackhole-s is honorable, but why do you have two method calls per @Benchmark? Wouldn't you want to quantify the performance of each particular invocation in isolation? Otherwise you risk to mask the performance improvement in one method with the performance degradation in another one. Splitting the methods would also help to use the implicit Blackhole-s (i.e. returning the result from @Benchmark instead of using the explicit Blackhole).

3. The benchmark fails to prevent call arguments from being predictable, which may subject some of the tests for constant folding inside the callee. See e..g JMHSample_10_ConstantFold.java.

But the largest problem, really, is blindly trusting the numbers. Instead of going on StackOverflow and asking others to use their magical vision to guess what could go wrong, you should actually check if anything have gone wrong! That would be the part of the larger question: "Why the performance of these benchmarks are different from each other?" Once you teach yourself asking this question for every benchmark, you will embrace the Dao of Benchmarking: you should benchmark for explanations, not for the numbers.

To answer those questions, you can either try to:

• Build the believable model how computers work, and how your code interacts with computers -- that will usually entail juggling the benchmark parameters and see if it responds in the ways your current model predicts it should respond (see how natural sciences handle the Nature in this regard).

• Look inside the computer operation to see what it is busy with. Use profilers, Luke! Now, for the nanoscale benchmarks you are having in your test, not all profilers are good. You will probably have to get down to generated assembly and/or hardware counters. Look around for one. JMH ships with a very basic -prof perfasm.

In other words, explore!