Benchmarking mistakes, part three

Part three of my beginner-level series on how to write bad benchmarking code can be found here.


7 thoughts on “Benchmarking mistakes, part three

  1. Mistake #7?

    Using a random number generator in benchmarking with a random seed.

    When testing two sets of code, or making subsequent test runs on the same code, using the same set of “random” values each time is desirable so that both runs process the same values. You can do this by specifying the seed to the RNG.

      • “Not knowing the upper bound” is another mistake. But also a feature for you, Eric. – Oh, I’ll cover that in in part n+m. =)

    • Depending upon the algorithm, I would think using a random seed would sometimes be a good thing and sometimes not. If a given sequence of random numbers will end up yielding the same general sequence of operations in the algorithms being tested, using a fixed seed may be good. On the other hand, if one is trying to compare the relative performance of e.g. two very different sorting algorithms, it would be possible that a fixed seed might consistently yield a sequence that usually favors one or disfavors the other. A random seed sometimes yield such sequences, of course, but a fixed seed that did so consistently would pose a bigger problem. I would suggest that even if one uses random seeds, one logs the values that are actually used to allow particular results to be reproduced (e.g. if one run of a program is excessively slow, retrying it with the same seed would reveal if it was an algorithmic problem or a system hiccup).

      • Sending random inputs to the algorithm is a type of fuzzing. Although it’s usually applied in security testing, it could apply quite well in performance testing (although I’ve never heard anyone mention it in that context). In both cases, the goal is to eliminate the biases that can be present in hand-written tests.

        • Normally, when doing such testing, you will still record the seed. If you are using truly random data (e.g., from, you could store the associated generated entropy, or wrap your random generator and have the ability to record and replay random numbers.

  2. “Because computer programmers love to verb words”…

    So meta, man. So meta. I love it. You verbed us good!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s