Persisting JMH Results for Regression Detection



JMH is a tool to measure Java code that is executed within  micro-second time scales. You may ask the question why do I care about the performance of code that by definition is executed in a very short period of time? It is important to only optimise code that will have a significant effect on the overall application. Sometimes a small portion of the your application's code is executed many times or your latency requirements of your application is in the micro-second time scale.

Once your application is performing as required it can be difficult to maintain that performance whilst making code changes. An automated collection of micro-benchmarks running on key components of your application is an invaluable tool in maintaining the required performance. As a starting point a history of your JMH results needs to be stored in order to analyse trends in benchmark results.

Influx JMH reporter is an open source tool we have written that parses the JMH JSON results file from a benchmark and publishes it to an InfluxDB host. We are then able to graph and analyse this data over time in order to detect regressions.

At the end of each JMH run execute:

java -jar influx-jmh-reporter-all.jar -u "http://influxhost:8086" -r results.json

This will persist the results in InfluxDB :
> show measurements
com.mycompany.mybenchmarks.PerformanceTestSuite.myBenchmark
> select * from "com.mycompany.mybenchmarks.PerformanceTestSuite.myBenchmark"
githash hostname       jvm     jvmArgs mode    score   scoreError  scoreUnit
------- --------      ---     ------- ----    -----   ----------  ---------
 7hf3k   myserver /bin/java  thrpt 28.4726       2       ops/us 
...


Stored data is monitored to detect a degradation in performance. Automated detection can be challenging as it requires isolating environmental factors in order to improve test measurement consistency.  JMH goes a long way however additional techniques such as reducing scheduling jitter (see Reducing System Jitter by Mark Price) goes much further.







Comments

Post a Comment

Popular posts from this blog

Obtaining the time in Java with nano-second precision

Image
At TransFICC for MiFID compliance (as explained in my previous blog) we need to obtain the time of an event within 100 microsecond accuracy with respect to UTC. To achieve this in Java we first need to be able to obtain the time from the OS with better than millisecond precision. We can then make use of Precision Time Protocol (PTP) to ensure the system clock is synchronised to UTC with sufficient accuracy. Take a look at the excellent posts from Luke Bigum on how to achieve this . As of Java version 8 it is not possible to directly obtain the time with higher than millisecond precision. Java 8 System.currentTimeMillis() will return the time since the Unix epoch in milliseconds. Higher precision time is possible on some underlying systems such as Linux. On Linux the code that the OpenJDK JVM executes is: The full source can be found  here . The code makes the function call gettimeofday(). This returns the time since the Linux epoch in micro seconds however it is rounded t...
Read more

Cracking the Nut - Open Sourcing from a Single Source Tree

Why a single source tree ? At a previous employer the code started out with the main code base in a single checkout with only infrastructure components of the code in their own repositories. As the code base grew the build time gradually become painful. To resolve this developers began to pull out components into their own repositories. This initially solved the problem of slow build times and improved productivity for the developers working on a single component. However, it soon became apparent that making changes to components that had many internal dependencies was particularly cumbersome. It also made it difficult to make changes across multiple repositories. This was solved by moving everything back into a single source tree. What about the slow build times? The short answer is Buck - a build tool from Facebook which is inspired by Google's internal build tool called Blaze, now open sourced as Bazel . In summary these build tools allow you to easily modularize your rep...
Read more