Living with CruiseControl and MSTest

A while ago I posted on getting  CruiseControl.NET working with MSTest. This has been working OK for us, but as with all things you learn a lot from living with it. So here are some tips and comments about our experiences:

Watch your timeouts

Many problems we have had have been due to steps in the build process taking longer than the timeout specified in the ccnet.config file. If this occurs the CruiseControl build process aborts as a failure, but here is the rub, the step that is taking too long still completes and so usually when you look at the logs for the failed build you see that the compile is OK, but the next step, in our case the tests are never run.

All very confusing until you check the raw build log file (on the left hand side menu of the actual build results).

TIP:  If all is going strange - up the timeouts after checking the raw log file.

Cleaning up after yourself

In my previous post I mentioned the problem that MSTEST tends to generate a lot of files with name of its choice (based on the machine, user and date) and you have to force it to write the result file with a fixed name so that it can be picked up by the CruiseControl logging system.

To add complexity if this fixed file already exists then the MSTEST task fails, so it has to be deleted with batch file before the testing is run.

Now when we had timeout issues (as mentioned above) we had the delete step after the build step and before actual test run. This had the effect that even though the build timed out and no test was run the old results file was still there and shown on the result page - all very confusing.

To get round this we moved the delete task before the build in the ccnet.config <tasks> section.

TIP: Do all your deletes and house keeping as soon as possible in case of unexpected build termination.

Results order

In our VS.NET 2005 projects we have code analysis switch on. This is in effect just FXCop built into the MSBUILD process. Now if you are using FXCop as a task in it own right in CruiseControl the results are shown in their own block, but if code analysis is used the results are displayed as warnings in the main compile process. This can have the effect that you get a successful build with a lot of warnings - all these warnings of cause need addressing but can tend to persist during the development phase.

Now in my previous post I had the MSTEST results appearing after the build results (and hence warnings). It turns out it is a right pain to have to page down through loads of warnings to see which test caused the MSTEST task to fail.  However, this is easily addressed, alter the dashboard.config file and move the MSTestSummary.xsl line above the compile.xsl as shown below:

Now this might seem strange, tests before build? However the way CruiseControl works is that if the build tasks fails then the tests task are not run, so the test results are blank (as long as you deleted any old result files first!) so the test result block is not shown, the compile results (with the errors) are shown at the top of the page.

TIP: Think about the results as to what is most important for you, put what you want to see most of the time at the start of the list.

The Future

In the longer term we are looking at Microsoft Team Foundation Server to provide a similar service as CruiseControl, giving a a better end-to-end a solution with Visual Studio.

Now  in effect, behind the scenes, this be a MSBUILD solution, it is an interesting point to consider as whether moving over to a MSBUILD solution within CruiseControl (as opposed to using Visual Studio builds via devenv.com) would be a good way to move forward.

Expect to see more posts on the subject in the future, and maybe a proposal for a session at the next DDD event.