Analyzing test code

Continuing with my series of answers to questions that were asked during my webcast last week…

Do the “checkers” (algorithms that find specific defect patterns) find defects in unit testing code?

If you want them to, yes.

First off, as I described in my talk, we deduce which code you want analyzed by monitoring the process that makes a clean build: by watching every invocation of the C# compiler during a clean build we know precisely which source code is being compiled, which assemblies are being referenced, and so on. If you don’t want your test code analyzed, don’t build it while the build capture system is running, and it won’t be analyzed. If you do want your test code analyzed, do build it.

Second, we have a feature called “Components” which allows you to separate defects sources into different logical groupings. This way in our defect presentation tool you can say “I only want to see defects found in the testing component”, and so on.

There are definitely some subtleties to consider when looking for defects in test code. For example, if you have a test which contains:

while(i++ < 10);
{ 
  Test.Assert(Bar(foo));
}

then that test has a genuine “stray semicolon” defect. (Of course the C# compiler will note this one as well, but that’s not germane to my point. The point is that this is clearly a case where the test is not behaving as desired and an analysis tool can determine that.)

By contrast, I’ve seen test code like this:

  Test.ExpectException<ArgumentNullException>(Bar(null));

The static analyzer will likely report that Bar is being called improperly, but that’s the whole point. It can be tricky to detect that this is an “intentional defect” and suppress it. Testing code tends to have a high concentration of intentional defects.

About these ads

3 thoughts on “Analyzing test code

  1. I can see how test code can be designed to contain constructs that would normally be considdered a bug, so I would imagine you would like to disable some checks for testcode. This could be a tough call as it could also possibly miss some actual defects. What is the practicle way you recomend to manage this?

    A more interesting question woulde be, do you have (or intend to have) checkers that are specifically designed to find errors that are common in test code, but not so in ‘normal’ code? Like a check for a test that does not perform any reasonable Assert on a tested methods returnvalue. I can imagine it is harder to draw the line here, but it would be interesting to hear how you think about this.

  2. Pingback: The Morning Brew - Chris Alcock » The Morning Brew #1656

  3. Pingback: Dew Drop – July 22, 2014 (#1819) | Morning Dew

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s