The blogs of Black Marble staff

Living with CruiseControl and MSTest

A while ago I posted on getting  CruiseControl.NET working with MSTest. This has been working OK for us, but as with all things you learn a lot from living with it. So here are some tips and comments about our experiences:

Watch your timeouts

Many problems we have had have been due to steps in the build process taking longer than the timeout specified in the ccnet.config file. If this occurs the CruiseControl build process aborts as a failure, but here is the rub, the step that is taking too long still completes and so usually when you look at the logs for the failed build you see that the compile is OK, but the next step, in our case the tests are never run.

All very confusing until you check the raw build log file (on the left hand side menu of the actual build results).

TIP:  If all is going strange - up the timeouts after checking the raw log file.

Cleaning up after yourself

In my previous post I mentioned the problem that MSTEST tends to generate a lot of files with name of its choice (based on the machine, user and date) and you have to force it to write the result file with a fixed name so that it can be picked up by the CruiseControl logging system.

To add complexity if this fixed file already exists then the MSTEST task fails, so it has to be deleted with batch file before the testing is run.

Now when we had timeout issues (as mentioned above) we had the delete step after the build step and before actual test run. This had the effect that even though the build timed out and no test was run the old results file was still there and shown on the result page - all very confusing.

To get round this we moved the delete task before the build in the ccnet.config <tasks> section.

TIP: Do all your deletes and house keeping as soon as possible in case of unexpected build termination.

Results order

In our VS.NET 2005 projects we have code analysis switch on. This is in effect just FXCop built into the MSBUILD process. Now if you are using FXCop as a task in it own right in CruiseControl the results are shown in their own block, but if code analysis is used the results are displayed as warnings in the main compile process. This can have the effect that you get a successful build with a lot of warnings - all these warnings of cause need addressing but can tend to persist during the development phase.

Now in my previous post I had the MSTEST results appearing after the build results (and hence warnings). It turns out it is a right pain to have to page down through loads of warnings to see which test caused the MSTEST task to fail.  However, this is easily addressed, alter the dashboard.config file and move the MSTestSummary.xsl line above the compile.xsl as shown below:

   <!-- plus any others you want to see -->


Now this might seem strange, tests before build? However the way CruiseControl works is that if the build tasks fails then the tests task are not run, so the test results are blank (as long as you deleted any old result files first!) so the test result block is not shown, the compile results (with the errors) are shown at the top of the page.

TIP: Think about the results as to what is most important for you, put what you want to see most of the time at the start of the list.

The Future

In the longer term we are looking at Microsoft Team Foundation Server to provide a similar service as CruiseControl, giving a a better end-to-end a solution with Visual Studio.

Now  in effect, behind the scenes, this be a MSBUILD solution, it is an interesting point to consider as whether moving over to a MSBUILD solution within CruiseControl (as opposed to using Visual Studio builds via would be a good way to move forward.

Expect to see more posts on the subject in the future, and maybe a proposal for a session at the next DDD event.

Upgrading this Blog from CommunityServer 2.0 to 2.1

I have just updated to the new release of CS2.1. All seemed straight forward, I unzipped the distribution, stopped my CS server, replaced the web home directory and then ran the cs_2.x_to_2.1_upgrade.sql script. This is all the readme said to do.

However when I restarted the server and tried to login I got the error (after editing the web.config to report detailed errors)

Procedure or Function 'aspnet_Profile_GetProperties' expects parameter '@TimeZoneAdjustment', which was not supplied.

I did a quick Google and it seems this is commonly seen when moving from .Net 1.1 to 2.0, but we were previously running CS2.0 (a new, default install from the MSI) on .Net 2.0, so it should not have been an issue.

The threads on the support forums at Telligent suggested running cs_ASPNET2_Membership_Schema_Update.sql which is also in the CS2.1 distribution. Some people were commenting running this if already run was a bad idea, but I thought it was worth a try before I rolled back to my CS2.0 backups. So I ran the script and and all seems OK - hence this post.

ps. I do like Windows Live Writer, great for writing a post over a period of time, irrespective of whether you have a connection or not.

New release of GUITester with Regular Expressions

Following my visit to Nxtgen I have added regular expression functionality to GUITester as promised.

The ClickTextTest attribute now has an optional extra boolean flag, this allows you to say if the text to match is a fixed string or a regular expression. The usage to check a textbox contains an email address:

[ClickTextTest("email test","textBoxRegularExpression",@"^\w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*$",true)]
private Button btnRegularExpression;
private TextBox textBoxRegularExpression;

There is also a simple text field test that allow you to validate that a field matches a regular expression, this is used in the form:

[RegularExpressionTest("email test",@"^\w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*$")]
private TextBox textBoxRegularExpression;

The new release can be found at, I hope this is of use to some people out there.

Sick of typos in your Blog

Do you find editing your blog on-line limiting whether it be:

  • awkward to use
  • does not have features you would like to see such as spelling checks
  • you can lost that big blog posting if there is comms failure to your ISP
  • or just that you want to create entries when you are not on-line such, as when commuting

If so have a look at Windows Live Writer, this is a beta of a desktop application that provides WYSIWGY authoring to most common Blog services, including Community Server (as this blog is hosted on).

First impressions are very good.

SharpRobo - another way to test GUIs

SharpRobo is another interesting way to test your win forms application, it uses recorder (test classes) to allow user operations to be replayed.

The potential issue for me is that you need to derive your forms and controls from SharpRobo base classes, it remains to be seen how much this is a limitation, but will worth a good look to see if it can work on  GUI projects

Black Marble @ the Coventry NxtGen User Group

I spoke at NxtGenUG’s Coventry user group earlier this week on GUI testing. All seemed to go well, but we shall see when I get the attendee feedback. It was good to see such a good turnout for a summer event, I know how hard it is to get people out at this time of year – I would hope that it was the scintillating subject I was speaking on , but I doubt it, more down to the hard work of the NxtGenUG organisers. It seems to be a user group going somewhere, I hear it is just about to open it third group down in Oxford.

On the subject of my presentation, though it was a rerun of the one I gave at DDD3 I can feel my GUI testing talk moving away from GUI specifics and becoming more of a general ‘hey have you thought of using attributes for meta data to better enable testing?’. I really like the idea of encouraging companies to view testers as developers not people who just tick boxes on a test protocol form. In fact when you add in using reflection and custom attributes the tester-developer gets to do some of what I consider the most interesting and intellectually changeling developing around.

As usual the audience managed to address one of the ongoing problems with the GUI tester, this time it was the ‘how do you provide dynamic data to a statically compiled test definition?’. I must thank Dave McMahon for suggesting a halfway house solution of allowing the test’s expected value to be defined as a regular expression, thus allow a far more flexible approach to test data. I am currently working this into the framework, expect a new version a few days.

If anyone who was at the meeting wants the slides I would point you to the version released from DDD3, there is only one slide different and that details the gotta’s for VS.Net 2005 testing projects, and I have blogged on that in more details on this site

Thanks again to everyone at NxtGenUG for a most enjoyable evening.

Sharepoint 2007 and SQL Express

I have just been building a couple of Sharepoint 2007 test servers with SQL Express

I have been seeing the error message

ReporterServer$SQLExpress $ is an illegal charactor.

having removed SQL Express and replaced it with SQL Server the error still persists

however after some gnashing of teeth and toil , the answer is that SQL Express reporting services creates an App Pool of the Name Reporter$SQLExpress which when removed allow Sharepoint to install applications