DDD7 Agenda Published

Well the votes are in and my proposed session for DDD7 on automated testing did not make the cut, but thanks to anyone who voted for it. I can’t say I am surprised that I am not on the list given the larger number of very interesting sessions proposed.

However, I am a little disappointed that even though there were a good percentage of the proposed sessions on testing related subjects only Ian’s on TDD and  Ben’s on Pex made it through; I really had expected to see Gojko’s on Fitnesse.Net on the list.

Given that much of the last Alt.Net conference was focused on acceptance testing the relative lack of testing related sessions surprised me. There seems to be a difference in interests between the DDD voting community and the Alt.Net attendees. Does this mean that the average person attending (or at least voting) for DDD sessions does not care about testing or thinks they have nothing to learn? or have I missed something about the nature of the two events?

Running fitness.Net tests in unit test – some tips

I posted a while ago on wiring in Fitness.Net into a unit test framework using HTML files to hold the tests. Well I have been using the technique for some workflow acceptance testing and hit a couple of gotta’s that are easy to forget:

  • Make sure the HTML files containing the user story tests are set to copy to the project output directory in the IDE – if they are not then the framework cannot find the tests, so obviously none are run. The danger is you think the problem is an incorrect class or method name, when it is a simple missing file problem.
  • If you edit the user story HTML file make sure you rebuild the solution. If you don’t do this the copy to the output directory might not be triggered as a simple build maybe skipped as the IDE will not see any changes to source files it needs to compile the project. This is especially easy to forget if you are using a test add-in such as Testdriven.net as opposed to clicking on build yourself.
  • Be careful with the HTML editor you use to create user story file in case it reformats the page. This is important for parameters (but it seems not for the method name fields). You need to make they are formatted

           <td>My Value</td>

as opposed to

           <td>
           My Value
           </td>

as the latter will include the carriage returns characters in the parameters passed into the test so will probably fail.

I think that is all for now, I will post any others I find as they crop up

Testing Driven Development for Workflow Foundation

As we move into the SOA world workflows will become more common and so the need to test them will increase. If we take the most simplistic view these are a sets if statements and loops so should be amenable to automated testing and TDD.

However you have to careful how you try to do this, it is too easy to forget that your workflow will run the the WF thread. Why is this problem?

Consider this scenario, if you create a simple sequential WF console application you get a block of code like this in the Main(args) method

using (WorkflowRuntime workflowRuntime = new WorkflowRuntime())
{
   AutoResetEvent waitHandle = new AutoResetEvent(false);
   workflowRuntime.WorkflowCompleted += delegate(object sender, WorkflowCompletedEventArgs e)
   {
      // could put asserts here (A)
      waitHandle.Set();
   };

   workflowRuntime.WorkflowTerminated += delegate(object sender, WorkflowTerminatedEventArgs e)
   {
          // could put asserts here (B)
   
   waitHandle.Set();
   };

  WorkflowInstance instance = workflowRuntime.CreateWorkflow(typeof(WorkflowConsoleApplication1.Workflow1));
   instance.Start();

   waitHandle.WaitOne();
}

// could put asserts here (C)

This block of code will be the basis of each of your unit tests, but as indicated you could put your test asserts at one of three places

  • In the success delegate
  • In the fail/terminate delegate
  • After the workflow has completed

The best place is the third option, when the workflow has finished, even though you know in a test whether it should completed OK or failed. The problem with the previous two locations is that the anonymous delegate will be run in the WF worker thread. Interestingly, if the assert passes in this location all will be OK, but if it fails the thrown assert exception (the mechanism used in nUnit to trap failure and in other test frameworks to my knowledge) will be in the wrong thread and so will not be picked up by the test harness, so in effect the test runner stalls waiting for completion or an exception, neither of which arrive. This issue is not the the case if the asserts are done when the workflow completes. This does mean that you have to pass results out of the workflow, but this should not be a major issue as the results should be simple objects (workflow parameters) or checks on external (mock?) systems, such as file systems, DBs or Smtp servers.

Though not always standard practice in TDD, I think it is a good idea here to move the code in the above sample’s using statement into a static method in your test class; it is boiler plate that will probably be use in many tests. As tests are meant to readable I think a single line call to the RunWorkFlow() method is better than a block of repeated loop and delegate code. Remember you do have to be careful here, as you may need to pass in parameters and get a return value to get the end state of the workflow; the requirement will depend what your workflow does and how it interacts with other systems. It maybe you need more than one version of the RunWorkFlow() static method depending on what is under test e.g one to harvest output parameters from completed workflow runs and another to harvest inner exceptions that are inside the workflow terminated delegate.

So now there is not reason to not test your WF workflows – assuming you can mock out any external connections, and if you are using web services or IoC to talk to data stores this should be fairly easy. 

Next I am off to try to do the same with SharePoint workflows.

Rebuild of GUITester

I have eventually got round to rebuilding my GUITester on Codeplex for VS2008. I must get round to looking to see if a similar declarative test markup system has any mileage for XAML. Something was talking about at Alt.net in the spring – how time flies.

Running Fitnesse.NET tests using MSTest

Update 29 Mar 2010 – See this post for some updated usage notes 

I have been looking at Fitnesse.NET for a while; if you have not come across this testing tool I cannot recommend Gojko Adzic book highly enough; an excellent real world introduction to Fitnesse.NET.

For me, the problem with Fitnesse is that it’s WIKI architecture does not lend itself to working within our Team Foundation Server based development model. However, as I am not really looking for a tool to allow Business Analysts to edit a central repository of tests the WIKI was not that important; so I have been looking at a way to store the Fitnesse tests inside a Visual Studio solution, so they are managed like a unit test.

The first step was to get my head round using Fitnesse without a WIKI. This is achieved using FolderRunner,exe that ships as part of Fitnesse.NET. This allows you to define a test as a HTML file as opposed to a WIKI page. Firstly I thought you could just copy the relevant contents.txt page out of the WIKI structure, but this does not work as this file holds the test not as a HTML table, as required, but in the WIKIs own format using | characters to define the table cells. So you have to create the test file containing the Fitnesse tables using your HTML editor of choice. Once you have done this you run the test using the following command line.

..\fitnesse\dotnet2\FolderRunner.exe -i userstory1.htm -a MyAssembly.dll -o results

A HTML results file ends up in the results directory and you get a simple results test count on the console.

All this is very good but how can you bolt this into Visual Studio? After a bit of thought I decided that the easiest option was to put the Fitnesse calls within a wrapper unit test. I chose MSTest as I was aiming to run the final solution within TFS Build (but any type if unit testing framework such as nUnit or mbUnit would have done).

Next I decided to make the required calls to run the tests in C#, calling the methods in the Fitnesse DLLs. This was as opposed to trying to shell out and run the FolderRunner.exe.

The easiest way to find the correct calls was to use Reflector to have a look at the code in the FolderRunner.exe (and the other Fit assemblies). After a brief splunk around I found the calls were fairly simple, so this is the process I ended up with to run the tests via MSTest:

  • I started with a solution that contains an application assembly and an second assembly that had all the methods required for the Fit tests (and made sure it all worked via the WIKI and/or command line FolderRunner)
  • Next I added a test project to the solution
  • In the test project I add a reference to the assembly that contains fit test methods and also the fit.dll. I had some problems here that when I ran the test, I got a The location of the file or directory ‘(path omitted)\fit.dll’ is not trusted. error. If you get this error follow the process on Don Felker’s blog to fix it This blog post seems to have been removed, in summary the post said:
    • If you download a DLL from the Internet, or get it in an email or where ever and you saved it to your disk (including a DLL in a zip too) it has some extra info attached to it called an “AES” file.
    • To fix this annoying issue, go to the DLL in Windows Explorer, right click to view the properties and then click the “Unblock” button at the bottom of the General panel

  • Add an HTML file to the Test Project and in it create the Fit test tables.
  • Set the HTML file properties so it is copied to the output directory.
  • Create a new Test Method to run all the tests in a the HTML file as shown below (obviously you will need to make sure the paths are right to the HTML files and assemblies for your project).

[TestMethod]
public void RunFitnessTests()
{
   fit.Runner.FolderRunner runner = new    fit.Runner.FolderRunner(new fit.Runner.ConsoleReporter());
   var errorCount = runner.Run(new string[] {
       “-i”,@”userstory1.htm”,
       “-a”,@”MyAssembly.dll”,
       “-o”,@”results”
});
    Assert.AreEqual(0, errorCount,runner.Results);
}

  • So now you can make sure this test works using TestDriven.NET to run this test, this checks the test works when the application is built to the standard bin\debug directory.
  • However this does not mean it will work with MSTest under the Visual Studio test runner. This is because, by default, the HTML file that holds the test will not be deployed to the TestResults directory (also the actual application assembly under test will not be copied). This is critical, as you have to be careful here, as if the HTML file is missing you get a false success. The Runner.Run method will return 0 as no test failed (as there were none defined) so the assert passes. To get round this you need to add the HTML file (and any other files required) to the list of files to deploy prior to testing. This is done by editing the deployment file list via the menu option Test | Edit Test Run Configuration | Deployment tab. An option to guard against this issue is to add an extra assert to make sure at least some tests were run and we don’t have four zeros in the results string, a bit of RegEx will do (this line is not in the sample project by the way)
           Assert.AreEqual(false,Regex.IsMatch(runner.Results, “^0.+?0.+?0.+?0.+?$”),”No tests appear to have been run”);

  • Once this is all done you can run the test via the Visual Studio Test menu option and you should see the test run and get a simple pass or fail result with a count of any failures in the error message. (Another possible option would be a make better use of the detailed HTML results file, at present we just use the simple result string for test results details)

So where do we end up?  I think we have a workable solution. We can run Fitnesse tests as easily as any unit test, and as with unit tests the Fitnesse test files are under TFS source control and kept in sync with the application they test. All without the need for the WIKI.

I have packaged up a sample of this solution and posted to this server.

CruiseControl & MSTest from Visual Studio 2008

Ages ago I posted on using MSTest and CruiseControl .NET with VS2005. As I am presenting tomorrow to the Yorkshire Extreme Programming Club on CC.Net I thought it a good idea to revisit this subject with VS2008.

Well basically nothing has changed, the old ccnet.config I detailed still works. However, I discovered that you no longer really need the block to delete the TestProject.TRX file as it seem the 2008 MSTest.EXE can overwrite an existing test results file.

However, we don’t get away without any changes. The one area that has changed is the format of the MSTEST results file. The MSTestSummary.XSL and MSTestReport.XSL files shipped with ccNet 1.3 which are used to build the Web Dashboard just give blank pages. I also checked the 1.4 Beta of CCnet and this also has the same version of XSL files.

Now a few people have posted on this problem, but up to now there appears to have been no public XSL files for the current V9.x version of MSTest results files. To address this problem Robert Hancock at Black Marble edited the MSTESTSummary.xsl file and using his work I managed to sort the MSTESTReport.xsl file. Both of these files I have posted on this server in a single zip. Once these files are copied into the CruiseControl.Net\webdashboard\xsl directory the MSTest reports should leap into life. Obviously only after you have setup all the other publishing bits I dealt with in my previous post.