Running Typemock Isolator based tests in TFS 2010 Team Build
Updated 2nd March 2010: Altered the sample arguments see this post for more details
Updated 27th Jan 2010: Gian Maria Ricci, another Team System MVP has done some more work on this problems and posted on how to create a custom activity to address the problem.
I have been looking at getting automated builds running on TFS2010 that make use of Typemock Isolator. This is not as straight forward as you would expect.
The issue is that you have start Isolator’s mocking interceptor before you run any tests that use Typemock (and stop it afterwards). If you are running in the VS IDE this is all done automatically, but is not done as part of an MSBuild Team Build Process by default.
In prior versions of TFS a solution was provide by Typemock in the form a pair of MSBUILD Tasks to start and stop the mocking process which you wired into your team build definition file, something like the example below
The problem with 2010 is that you no longer get this single MSBUILD file to manage the Team Build, it is now a XAML workflow. I know I could use the model for legacy support of the older build process, but where is the fun in that?
Attempt 1 – Adding more MSBuild projects
My first idea was to add a MSBUILD activity just before the MSTEST activity block (and another after to stop the mocking interceptor) and pointed it at a Typemock.Proj file (placed in in a suitable location) and calling the correct target. The proj file is shown below
In the build logs I could see the activities ran, and a typemock.log file is created for the MSBuilds (note a problem here that as I use the same proj for both the start and stop build target the log file gets over written, so a good idea is to pass in explicit log file names for each call to MSBUILD). However, even though everything seemed to run OK the test that need Typemock fail with the error
Test method CallTracker.BusinessLogic.Tests.PrintTests.PrintAllCalls_TypeMockedPrinter_2CallsPrinted threw exception:
TypeMock.TypeMockException:
*** Typemock Isolator is not currently enabled.
To enable do one of the following:
* To run Typemock Isolator as part of an automated process you can:
- run tests via TMockRunner.exe command line tool
- use 'TypeMockStart' tasks for MSBuild or NAnt
* To work with Typemock Isolator inside Visual Studio.NET:
set Tools->Enable Typemock Isolator from within Visual Studio
For more information consult the documentation (see 'Running' topic)
This is exactly the error you expect if you have not started the mocking interceptor.
After a chat with Typemock support I understood the problem. In the TFS 2005/8 model we had a single MSBUILD process running. It compiled the code, ran the tests, it did everything. So if we started mocking interception in this process everything was good and it all worked. Under the 2010 model we have the new XAML based process, this starts a MSBUILD process to do the compile, and in our case another to start mocking, then MSTest to run and finally another MSBUILD to stop mocking. So we have at least four process within the build and the reality is we have only switched on mocking for the duration of the MSBUILD task calling the StartTypemock task.
Attempt 2 – Other task types
I did also look at starting mocking using the INVOKEMETHOD task and by calling the Typemock provided batch files. These all resulted in the same basic problem, mocking was not switched on when MSTest ran.
A solution
The solution is to use the TMockRunner.EXE that is shipped with TypeMock Isolator. This is a wrapper EXE that runs a command line tools passing in the required set of parameters. All the TMockRunner does is, as the name suggests, start and stop the mocking as needed.
To get this to work I removed the MSTest task in the build workflow and replaced it will a InvokeProcess one with the following parameters
FileName: "C:Program Files (x86)TypemockIsolator6.0TMockRunner.exe"
Arguments:
"""C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEMSTest.exe"" /nologo /testcontainer:""" + String.Format("{0}BinariesTestproject.dll", BuildDirectory) + """ /publish:""http://typhoon:8080/tfs/DefaultCollection"" /publishbuild:""" + BuildDetail.Uri.ToString() + """ /teamproject:""" + BuildDetail.TeamProject + """ /platform:""Any CPU"" /flavor:""Debug"" /resultsfile:""" + String.Format("{0}BinariesTest.Trx", BuildDirectory) + """ "
"""C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEMSTest.exe"" /nologo /testcontainer:""" + String.Format("{0}BinariesBusinessLogic.Tests.dll", BuildDirectory) + """ /publish:""http://typhoon:8080/tfs/DefaultCollection"" /publishbuild:""" + BuildDetail.Uri.ToString() + """ /teamproject:""" + BuildDetail.TeamProject + """ /platform:""" + platformConfiguration.Platform + """ /flavor:""" + platformConfiguration.Configuration + """ /resultsfile:""" + String.Format("{0}BinariesTest.Trx", BuildDirectory) + """ "
A few things to note on the arguments and settings
- The arguments have to appear as a single string, so this is why you have all the double quotes escaping
- The Resultsfile is dumped into the binaries directory as this is cleaned out on each build. It has to go in an existing directory (we have no TestResults by default, of course we could create one, but why go to the effort?). This is a simple solution as the MSTest still seems to fail if the Test.TRX file already exists, Note I did try to use the /resultsfileroot parameter (which is used by default by the 2010 MSTest task) but this is not support via the command line.
- I would like to pickup the publish URL, platform and flavor automatically, I need to dig port into the TFS API to find these values at build time.
- I have not wired the MSTest output back into the main log, so it is hard to see what has happened without just looking for the creation of the TRX file.
- Watch out for DCOM activation errors. There is a good chance the user you are running the build process as does not have the right to run anything else. If your InvokeProcess seems to do nothing check the build PCs error log for DCOM activation errors and either give the build user the correct rights or swap to a user that already has the rights.
Once this was all done I could do my build and the test ran and passed.
The one remaining issues was to set the Test status for the build base done these results, I have no solution to that as yet, but I am still looking