Benchmarking mistakes, part one

The good people over at Tech.Pro have asked me to write a short series of articles for their site on the subject of mistakes I've made and seen others make when writing benchmark performance tests of C# code. The series will be pitched at the beginner level.

You can check out part one here. Enjoy!

13 thoughts on “Benchmarking mistakes, part one

    • Wow - I never knew about `Process.TotalProcessorTime`. Seems like for performing benchmarks this might be ideal - especially in the world of power saving devices (like laptops) where we might be getting benchmarks clouded by power optimizations without necessarily realizing it. Interesting about the stopwatch being unavailable in some cases - I was definitely not aware of this.

      Nice link, Jon!

    • Oh yes, that StopWatch thing!
      I've once wasted some hours looking for an assumed bug, but in the end it was StopWatch fault.
      On our development and test servers all was well, but on production it looked like our Cache needs 12 seconds (sometimes only) instead of a few milliseconds.
      The solution was found as the StopWatch was compared to DateTime.Now - there simply was no slow cache, but the StopWatch was simply insane.
      Never had expected this :-(

  1. I doubt that the timing thing described in that four and a half year old blog post is still an issue. We ran into this in 2005 or 2006 on a dual-core Athlon, and Microsoft had a fix for it. See my article, "Timer Troubles" at http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=474.

    I strongly suspect that it's a non-issue by now if you have modern hardware and/or an operating system newer than Windows XP. Everything I've read on the subject leads me to believe that QueryPerformanceCounter and QueryPerformanceFrequency work correctly in the face of variable clock rates and multiple CPUs. If Stopwatch is implemented in terms of those two functions, then this "problem" has been dead for five or more years.

    • Here the problem with Stopwatch occured on Windows 2003 Server (fully patched, serverworks board with an Opteron).
      Another problem are virtual machines on W2008R2, where the clock is slower than reality - I found no good solution up to now. The Host time is fine all the time.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>