Wednesday, March 30, 2011

The complexities of interpreting a benchmark

Benchmark wars are no fun, and I hate them as much as the next engineer. So it's a breath of fresh air to read, over at the Real World Technologies web site, David Kanter's thorough and detailed analysis of the recent benchmarks of the new AMD server micro-architecture code-named "bulldozer".

It's worth reading Kanter's article just to get some perspective into how hard it is to conduct a detailed and accurate analysis of a benchmark of a modern server. Kanter identifies dozens of variables that can affect the overall results, both procedural aspects such as compiler settings and runtime options, as well as engineering decisions such as system bus speeds, memory subsystem design, power management, etc.

Kudos to Kanter for taking the time to study the issues in such detail, and for sharing with us the thought process he went through as he analyzed the data that has emerged so far. For my part, I'm much less interested in the results than in the process, as it's always good to consider how one can become a better benchmarker, and reading a benchmark critique is a great way to do that.

No comments:

Post a Comment