JVM provides dozens of configuration flags, with many flags intended
for tuning application performance. We empirically study the impact of
JVM configuration flags on software testing runtime. We focus on an
extensive study that shows not only the great impact of JVM
configurations on test runtime (up to 43.89% reduction in runtime when
using certain configurations) but also shows that those configurations
that reduce runtime are rare and thus hard to find. Modern techniques
based on machine learning or combinatorial testing that search through
combinations of configuration flags are still not as effective at
finding the best configurations for test runtime. Finally, we show
that JVM configurations that provide good speedup retain this power
over a number of commits. We believe that this paper provides strong
motivation for further work on finding the best JVM configurations to
optimize test runtime.