Pitfalls of Benchmarking on Modern Systems

What could it be? It could be that the operating system decided to give it different physical memory, or run it on a different kind of core (many CPUs these days have 2 or more different types of cores with different performance). Perhaps, we ran out of thermal budget and the CPU ran at a lower clock speed to avoid overheating? Or, since we are on a JVM, perhaps the compiler saw slightly different information for the types/behavior seen in the first and second iterations, and thus, made slightly different optimization decisions? This is possible because compilation happens on a background thread, and thus, even when the benchmark is deterministic, the used profiling information is to some degree racy. — Pitfalls of Benchmarking on Modern Systems by Stefan Marr In my PhD I’ve spent a lot of time doing careful benchmarking of parallel programs. During that time, I’ve learn a lot of variables that matter when doing this type of work: CPU, RAM, interpreter/compiler, room temperature,…