Empty methods noticeably slower in Java 11 than Java 8

Question

I was comparing the performance of JDK 8 and 11 using jmh 1.21 when I ran across some surprising numbers:

Java version: 1.8.0_192, vendor: Oracle Corporation

Benchmark                Mode  Cnt  Score   Error  Units
MyBenchmark.emptyMethod  avgt   25  0.362 ± 0.001  ns/op


Java version: 9.0.4, vendor: Oracle Corporation

Benchmark                Mode  Cnt  Score    Error  Units
MyBenchmark.emptyMethod  avgt   25  0.362 ±  0.001  ns/op


Java version: 10.0.2, vendor: Oracle Corporation

Benchmark                Mode  Cnt  Score   Error  Units
MyBenchmark.emptyMethod  avgt   25  0.723 ± 0.001  ns/op


Java version: 11.0.1, vendor: Oracle Corporation

Benchmark                Mode  Cnt  Score   Error  Units
MyBenchmark.emptyMethod  avgt   25  0.724 ± 0.002  ns/op

OpenJDK 11 and 12 perform similar to OracleJDK 11. I have omitted their numbers for the sake of brevity.

I understand that microbenchmarks do not indicate the performance behavior of real-life applications. Still, I'm curious where this difference is coming from. Any ideas?

Here is the benchmark in its entirety:

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>jmh</groupId>
    <artifactId>empty-method</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>JMH benchmark sample: Java</name>

    <dependencies>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-core</artifactId>
            <version>${jmh.version}</version>
        </dependency>
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>${jmh.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <jmh.version>1.21</jmh.version>
        <javac.target>1.8</javac.target>
        <uberjar.name>benchmarks</uberjar.name>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>1.4.1</version>
                <executions>
                    <execution>
                        <id>enforce-versions</id>
                        <goals>
                            <goal>enforce</goal>
                        </goals>
                        <configuration>
                            <rules>
                                <requireMavenVersion>
                                    <version>3.0</version>
                                </requireMavenVersion>
                            </rules>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <compilerVersion>${javac.target}</compilerVersion>
                    <source>${javac.target}</source>
                    <target>${javac.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <finalName>${uberjar.name}</finalName>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>org.openjdk.jmh.Main</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <!--
                                            Shading signed JARs will fail without this.
                                            http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
                                    -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <plugin>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>2.6.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-deploy-plugin</artifactId>
                    <version>2.8.2</version>
                </plugin>
                <plugin>
                    <artifactId>maven-install-plugin</artifactId>
                    <version>2.5.2</version>
                </plugin>
                <plugin>
                    <artifactId>maven-jar-plugin</artifactId>
                    <version>3.1.0</version>
                </plugin>
                <plugin>
                    <artifactId>maven-javadoc-plugin</artifactId>
                    <version>3.0.0</version>
                </plugin>
                <plugin>
                    <artifactId>maven-resources-plugin</artifactId>
                    <version>3.1.0</version>
                </plugin>
                <plugin>
                    <artifactId>maven-site-plugin</artifactId>
                    <version>3.7.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-source-plugin</artifactId>
                    <version>3.0.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.22.0</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

src/main/java/jmh/MyBenchmark.java:

package jmh;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class MyBenchmark
{
    @Benchmark
    public void emptyMethod()
    {
    }
}

Here is the Windows-specific script I use. It should be trivial to translate it to other platforms:

set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_192
call mvn -V -Djavac.target=1.8 clean install
"%JAVA_HOME%\bin\java" -jar target\benchmarks.jar

set JAVA_HOME=C:\Program Files\Java\jdk-9.0.4
call mvn -V -Djavac.target=9 clean install
"%JAVA_HOME%\bin\java" -jar target\benchmarks.jar

set JAVA_HOME=C:\Program Files\Java\jdk-10.0.2
call mvn -V -Djavac.target=10 clean install
"%JAVA_HOME%\bin\java" -jar target\benchmarks.jar

set JAVA_HOME=C:\Program Files\Java\oracle-11.0.1
call mvn -V -Djavac.target=11 clean install
"%JAVA_HOME%\bin\java" -jar target\benchmarks.jar

My runtime environment is:

Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T14:41:47-04:00)
Maven home: C:\Program Files\apache-maven-3.6.0\bin\..
Default locale: en_CA, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

More specifically, I am running Microsoft Windows [Version 10.0.17763.195].

`-prof perfasm` would tell you a more complete story. Or `-prof xperfasm` on Windows. — Aleksey Shipilev, Jan 02 '19 at 17:29

Aleksey Shipilev · Accepted Answer · 2019-01-05T22:49:05.607

You are measuring empty benchmarks, not empty methods. In other words, measuring the minimal infrastructure code that handles the benchmark itself. This is easy to dissect, because you'd expect only a few instructions on the hot path. JMH's -prof perfasm or -prof xperfasm would give you those hottest instructions in seconds.

I think the effect is due to Thread-Local Handshakes (JEP 312), see:

8u191: 0.389 ± 0.029 ns/op [so far so good]

  3.60%  ↗  ...a2: movzbl 0x94(%r8),%r10d
  0.63%  │  ...aa: add    $0x1,%rbp
 32.82%  │  ...ae: test   %eax,0x1765654c(%rip) ; global safepoint poll
 58.14%  │  ...b4: test   %r10d,%r10d
         ╰  ...b7: je     ...a2

11.0.2: 0.585 ± 0.014 ns/op [oops, regression]

  0.31%  ↗  ...70: movzbl 0x94(%r9),%r10d    
  0.19%  │  ...78: mov    0x108(%r15),%r11  ; reading the thread-local poll addr
 25.62%  │  ...7f: add    $0x1,%rbp          
 35.10%  │  ...83: test   %eax,(%r11)       ; thread-local safepoint poll
 34.91%  │  ...86: test   %r10d,%r10d
         ╰  ...89: je     ...70

11.0.2, -XX:-ThreadLocalHandshakes: 0.399 ± 0.048 ns/op [back to 8u perf]

  5.64%  ↗  ...62: movzbl 0x94(%r8),%r10d    
  0.91%  │  ...6a: add    $0x1,%rbp          
 34.36%  │  ...6e: test   %eax,0x179be88c(%rip) ; global safepoint poll
 54.79%  │  ...74: test   %r10d,%r10d
         ╰  ...77: je     ...62

I think this is largely visible mostly in tight loops like this one.

UPD: Hopefully, more details here.

Interesting indeed. Who else... Thanks for making an answer. — Naman, Jan 02 '19 at 18:24
Why does JMH not subtract the cost of an empty benchmark from the method being benchmarked automatically? — Stefan Reich, Nov 18 '19 at 12:57

Empty methods noticeably slower in Java 11 than Java 8

1 Answers1

Linked