Benchmarking mistakes, part one

The good people over at Tech.Pro have asked me to write a short series of articles for their site on the subject of mistakes I’ve made and seen others make when writing benchmark performance tests of C# code. The series will be pitched at the beginner level.

You can check out part one here. Enjoy!

About these ads

14 thoughts on “Benchmarking mistakes, part one

    • Wow – I never knew about `Process.TotalProcessorTime`. Seems like for performing benchmarks this might be ideal – especially in the world of power saving devices (like laptops) where we might be getting benchmarks clouded by power optimizations without necessarily realizing it. Interesting about the stopwatch being unavailable in some cases – I was definitely not aware of this.

      Nice link, Jon!

    • Oh yes, that StopWatch thing!
      I’ve once wasted some hours looking for an assumed bug, but in the end it was StopWatch fault.
      On our development and test servers all was well, but on production it looked like our Cache needs 12 seconds (sometimes only) instead of a few milliseconds.
      The solution was found as the StopWatch was compared to DateTime.Now – there simply was no slow cache, but the StopWatch was simply insane.
      Never had expected this :-(

  1. I doubt that the timing thing described in that four and a half year old blog post is still an issue. We ran into this in 2005 or 2006 on a dual-core Athlon, and Microsoft had a fix for it. See my article, “Timer Troubles” at http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=474.

    I strongly suspect that it’s a non-issue by now if you have modern hardware and/or an operating system newer than Windows XP. Everything I’ve read on the subject leads me to believe that QueryPerformanceCounter and QueryPerformanceFrequency work correctly in the face of variable clock rates and multiple CPUs. If Stopwatch is implemented in terms of those two functions, then this “problem” has been dead for five or more years.

    • Here the problem with Stopwatch occured on Windows 2003 Server (fully patched, serverworks board with an Opteron).
      Another problem are virtual machines on W2008R2, where the clock is slower than reality – I found no good solution up to now. The Host time is fine all the time.

  2. Pingback: If you’re using enum.ToString() that often, you’re doing it wrong | David Zych

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s