Lessons learnt building a custom activity to run Typemock Isolator in VS2010 Team Build
Updated 25th March 2010 - All the source is now available at the Typemock Add-in site
Updated 2nd July 2010 - Some usage notes posted
Updated 26th July 2011 - More usage notes
Updated 21st Nov 2011 - Typemock Isolator now has direct support for TFS 2010 Build, see usage notes
I have previously posted on how you can run Typemock Isolator based tests within a VS2010 using the InvokeMethod activity. After this post Gian Maria Ricci, a fellow Team System MVP suggested it would be better to put this functionality in a custom code activity, and provided the basis of the solution. I have taken this base sample and worked it up to be a functional activity, and boy have I learnt a few things doing it.
Getting the custom activity into a team build
Coding up a custom Team Build activity is not easy, there are a good few posts on the subject (Jim Lamb’s is a good place to start). The problem is not writing the code but getting the activity into the VS toolbox. All documentation gives basically the same complex manual process, there is no way of avoiding it. Hopefully this will be addressed in a future release of Visual Studio. But for now the basic process is this:
- Create a Class Library project in your language of choice
- Code up your activity inheriting it from the CodeActivity
class - Branch the build workflow, that you wish to use for testing, into the folder of the class library project
- Add the build workflow’s .XAML file to the class library project then set it’s properties: “build action” to none and “copy to output directory” to do not copy
- Open the .XAML file (in VS2010), the new activity should appear in the toolbox, it can be dropped onto the workflow. Set the properties required.
- Check in the file .XAML file
- Merge the .XAML file to the original location, if you get conflicts simply tell merge to use the new version discarding the original version, so effectively overwriting the original version with the version edited in the project.
- Check in the merged original .XAML file that now contains the modifications.
- Take the .DLL containing the new activity and place it in a folder under source control (usually under the BuildProcessTemplates folder)
- Set the Build Controller’s custom assemblies path to point to this folder (so your custom activity can be loaded)
[](/wp-content/uploads/sites/2/historic/image_659D8BC7.png)
- Run the build and all should be fine
But of course is wasn’t. I kept getting the error when I ran a build
TF215097: An error occurred while initializing a build for build definition Typemock TestBuildTest Branch: Cannot create unknown type '{clr-namespace:TypemockBuildActivity}ExternalTestRunner'.
This was because I had not followed the procedure correctly. I had tried to be clever. Instead of step 6 and onwards I had had an idea. I created a new build that referenced the branched copy of the .XAML file in the class library project directly. I thought this would save me a good deal of tedious merging while I was debugged my process. It did do this but introduced other issues
The problem was when I inspected the .XAML in my trusty copy of Notepad, I saw that there was no namespace declared for my assembly (as the TF21509 error suggested). If I looked at the actual activity call in the file it was declared as <local:ExternalTestRunner …… />, the local: replacing the namespace reference I would expect. This is obviously down to the way I was editing the .XAML file in the VS2010.
The fix is easy, using Notepad I added a namespace declaration to the Activity block
<Activity …… xmlns:t="clr-namespace:TypemockBuildActivity;assembly=TypemockBuildActivity" >
and then edited the references from local: to t: (the alias for my namespace) for any classes called from the custom assembly e.g.
<t:ExternalTestRunner ResultsFileRoot="{x:Null}" BuildNumber="[BuildDetail.Uri.ToString()]" Flavor="[platformConfiguration.Configuration]" sap:VirtualizedContainerService.HintSize="200,22" MsTestExecutable="C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEMSTest.exe" Platform="[platformConfiguration.Platform]" ProjectCollection="http://typhoon:8080/tfs/DefaultCollection" Result="[ExternalTestRunnerResult]" ResultsFile="ExternalTestRunner.Trx" SearchPathRoot="[outputDirectory]" TeamProjectName="[BuildDetail.TeamProject]" TestAssemblyNames="[testAssemblies.ToArray()]" TestRunnerExecutable="C:Program Files (x86)TypemockIsolator6.0TMockRunner.exe" TestSettings="[localTestSettings]" />
Once this was done I could use my custom activity in a Team Build, though I had to make this manual edit every time I edited the branched .XAML file in VS2010 IDE. So I had swapped repeated merges with repeated editing, you take your view as to which is worst.
So what is in my Typemock external test runner custom activity?
The activity is basically the same as the one suggest by Gian Maria, it takes all the same parameters as the MSTest team build activity and then executes the TMockRunner to wrapper MSTest. What I have done is add a couple of parameters that were missing in the original sample and also added some more error traps and logging.
1using System;
using System.Collections.Generic;
1using System.Linq;
using System.Text;
1using System.Activities;
using System.IO;
1using System.Diagnostics;
using Microsoft.TeamFoundation.Build.Workflow.Activities;
1using Microsoft.TeamFoundation.Build.Client;
using System.Text.RegularExpressions;
namespace TypemockBuildActivity
1{
public enum ExternalTestRunnerReturnCode { Unknown =0 , NotRun, Passed, Failed };
\[BuildExtension(HostEnvironmentOption.Agent)\]
1 \[BuildActivity(HostEnvironmentOption.All)\]
public sealed class ExternalTestRunner : CodeActivity<ExternalTestRunnerReturnCode>
1 {
// Define an activity input argument of type string
/// <summary>
1 /// The name of the wrapper application, usually tmockrunner.exe
/// </summary>
1 public InArgument<string\> TestRunnerExecutable { get; set; }
2```
3
4```
5 /// <summary>
/// The name of the application that actually runs the test, defaults to MSTest.exe if not set
1 /// </summary>
public InArgument<string\> MsTestExecutable { get; set; }
/// <summary>
1 /// The project collection to publish to e.g. http://tfs2010:8080/tfs/DefaultCollection
/// </summary>
1 public InArgument<string\> ProjectCollection { get; set; }
2```
3
4```
5 /// <summary>
/// The build ID to to publish to e.g. vstfs:///Build/Build/91
1 /// </summary>
public InArgument<string\> BuildNumber { get; set; }
/// <summary>
1 /// The project name to publish to e.g: "Typemock Test"
/// </summary>
1 public InArgument<string\> TeamProjectName { get; set; }
2```
3
4```
5 /// <summary>
/// The platform name to publish to e.g. Any CPU
1 /// </summary>
public InArgument<string\> Platform { get; set; }
/// <summary>
1 /// The flavour (configuration) to publish to e.g. "Debug"
/// </summary>
1 public InArgument<string\> Flavor { get; set; }
2```
3
4```
5 /// <summary>
/// Array of assembly names to test
1 /// </summary>
public InArgument<string\[\]> TestAssemblyNames { get; set; }
/// <summary>
1 /// Where to search for assemblies under test
/// </summary>
1 public InArgument<string\> SearchPathRoot { get; set; }
2```
3
4```
5 /// <summary>
/// A single name result file
1 /// </summary>
public InArgument<string\> ResultsFile { get; set; }
/// <summary>
1 /// A directory to store results in (tends not be used if the ResultFile is set)
/// </summary>
1 public InArgument<string\> ResultsFileRoot { get; set; }
2```
3
4```
5 /// <summary>
/// The file that list as to how test should be run
1 /// </summary>
public InArgument<string\> TestSettings { get; set; }
// If your activity returns a value, derive from CodeActivity<TResult>
1 // and return the value from the Execute method.
protected override ExternalTestRunnerReturnCode Execute(CodeActivityContext context)
1 {
String msTestOutput = string.Empty;
1 ExternalTestRunnerReturnCode exitMessage = ExternalTestRunnerReturnCode.NotRun;
2```
3
4```
5 if (CheckFileExists(TestRunnerExecutable.Get(context)) == false)
{
1 LogError(context, string.Format("TestRunner not found {0}", TestRunnerExecutable.Get(context)));
}
1 else
{
1 String mstest = MsTestExecutable.Get(context);
if (CheckFileExists(mstest) == false)
1 {
mstest = GetDefaultMsTestPath();
1 }
2```
3
4```
5 String testrunner = TestRunnerExecutable.Get(context);
6```
7
8```
9 var arguments = new StringBuilder();
arguments.Append(string.Format(""{0}"", mstest));
1 arguments.Append(" /nologo ");
2```
3
4```
5 // the files to test
foreach (string name in TestAssemblyNames.Get(context))
1 {
arguments.Append(AddParameterIfNotNull("testcontainer", name));
1 }
2```
3
4```
5 // settings about what to test
arguments.Append(AddParameterIfNotNull("searchpathroot", SearchPathRoot.Get(context)));
1 arguments.Append(AddParameterIfNotNull("testSettings", TestSettings.Get(context)));
2```
3
4```
5 // now the publish bits
if (string.IsNullOrEmpty(ProjectCollection.Get(context)) == false)
1 {
arguments.Append(AddParameterIfNotNull("publish", ProjectCollection.Get(context)));
1 arguments.Append(AddParameterIfNotNull("publishbuild", BuildNumber.Get(context)));
arguments.Append(AddParameterIfNotNull("teamproject", TeamProjectName.Get(context)));
1 arguments.Append(AddParameterIfNotNull("platform", Platform.Get(context)));
arguments.Append(AddParameterIfNotNull("flavor", Flavor.Get(context)));
1 }
2```
3
4```
5 // where do the results go, tend to use one of these not both
arguments.Append(AddParameterIfNotNull("resultsfile", ResultsFile.Get(context)));
1 arguments.Append(AddParameterIfNotNull("resultsfileroot", ResultsFileRoot.Get(context)));
2```
3
4```
5 LogMessage(context, string.Format("Call Mstest With Wrapper \[{0}\] and arguments \[{1}\]", testrunner, arguments.ToString()), BuildMessageImportance.Normal);
6```
7
8```
9 using (System.Diagnostics.Process process = new System.Diagnostics.Process())
{
1 process.StartInfo.FileName = testrunner;
process.StartInfo.WorkingDirectory = SearchPathRoot.Get(context);
1 process.StartInfo.WindowStyle = ProcessWindowStyle.Normal;
process.StartInfo.UseShellExecute = false;
1 process.StartInfo.ErrorDialog = false;
process.StartInfo.CreateNoWindow = true;
1 process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.Arguments = arguments.ToString();
1 try
{
1 process.Start();
msTestOutput = process.StandardOutput.ReadToEnd();
1 process.WaitForExit();
// for TypemockRunner and MSTest this is alway seems to be 1 so does not help tell if test passed or not
1 // In general you can detect test failures by simply checking whether mstest.exe returned 0 or not.
// I say in general because there is a known bug where on certain OSes mstest.exe sometimes returns 128 whether
1 // successful or not, so mstest.exe 10.0 added a new command-line option /usestderr which causes it to write
// something to standard error on failure.
// If (error data received)
1 // FAIL
// Else If (exit code != 0 AND exit code != 128)
1 // FAIL
// Else If (exit code == 128)
1 // Write Warning about weird error code, but SUCCEED
// Else
1 // SUCCEED
2```
3
4```
5 ///int exitCode = process.ExitCode;
LogMessage(context, string.Format("Output of ExternalTestRunner: {0}", msTestOutput), BuildMessageImportance.High);
1 }
catch (InvalidOperationException ex)
1 {
LogError(context, "ExternalTestRunner InvalidOperationException :" + ex.Message);
1 }
2```
3
4```
5 exitMessage = ParseResultsForSummary(msTestOutput);
}
1 }
LogMessage(context, string.Format("ExternaTestRunner exiting with message \[{0}\]", exitMessage), BuildMessageImportance.High);
1 return exitMessage;
}
/// <summary>
1 /// Adds a parameter to the MSTest line, it has been extracted to allow us to do a isEmpty chekc in one place
/// </summary>
1 /// <param name="parameterName">The name of the parameter</param>
/// <param name="value">The string value</param>
1 /// <returns>If the value is present a formated block is return</returns>
private static string AddParameterIfNotNull(string parameterName, string value)
1 {
var returnValue = string.Empty;
1 if (string.IsNullOrEmpty(value) == false)
{
1 returnValue = string.Format(" /{0}:"{1}"", parameterName, value);
}
1 return returnValue;
}
/// <summary>
1 /// A handler to check the results for the success or failure message
/// This is a rough way to do it, but is more reliable than the MSTest exit codes
1 /// It returns a string as opposed to an exit code so that it
/// Note this will not work of the /usestderr flag is used
1 /// </summary>
/// <param name="output">The output from the test run</param>
1 /// <returns>A single line summary</returns>
private static ExternalTestRunnerReturnCode ParseResultsForSummary(String output)
1 {
ExternalTestRunnerReturnCode exitMessage = ExternalTestRunnerReturnCode.NotRun;
1 if (Regex.IsMatch(output, "Test Run Failed"))
{
1 exitMessage = ExternalTestRunnerReturnCode.Failed;
}
1 else if (Regex.IsMatch(output, "Test Run Completed"))
{
1 exitMessage = ExternalTestRunnerReturnCode.Passed;
}
1 else
{
1 exitMessage = ExternalTestRunnerReturnCode.Unknown;
}
return exitMessage;
1 }
2```
3
4```
5 /// <summary>
/// Handles finding MSTest, checking both the 32 and 64 bit paths
1 /// </summary>
/// <returns></returns>
1 private static string GetDefaultMsTestPath()
{
1 String mstest = @"C:Program FilesMicrosoft Visual Studio 10.0Common7IDEmstest.exe";
if (CheckFileExists(mstest) == false)
1 {
mstest = @"C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEMSTest.exe";
1 if (CheckFileExists(mstest) == false)
{
1 throw new System.IO.FileNotFoundException("MsTest file cannot be found");
}
1 }
return mstest;
1 }
2```
3
4```
5 /// <summary>
/// Helper method so we log in both the VS Build and Debugger modes
1 /// </summary>
/// <param name="context">The workflow context</param>
1 /// <param name="message">Our message</param>
/// <param name="logLevel">Team build importance level</param>
1 private static void LogMessage(CodeActivityContext context, string message, BuildMessageImportance logLevel)
{
1 TrackingExtensions.TrackBuildMessage(context, message, logLevel);
Debug.WriteLine(message);
1 }
2```
3
4```
5 /// <summary>
/// Helper method so we log in both the VS Build and Debugger modes
1 /// </summary>
/// <param name="context">The workflow context</param>
1 /// <param name="message">Our message</param>
private static void LogError(CodeActivityContext context, string message)
1 {
TrackingExtensions.TrackBuildError(context, message);
1 Debug.WriteLine(message);
}
/// <summary>
1 /// Helper to check a file name to make sure it not null and that the file it name is present
/// </summary>
1 /// <param name="fileName"></param>
/// <returns></returns>
1 private static bool CheckFileExists(string fileName)
{
1 return !((string.IsNullOrEmpty(fileName) == true) || (File.Exists(fileName) == false));
}
1 }
}
1
2This activity does need a good bit of configuring to use it in a real build. However, as said previously, the options it takes are basically those needed for the MSTest activity, so you just replace the existing calls to the MSTest activities as shown in the graph below should be enough.
3
4[](/wp-content/uploads/sites/2/historic/image_3744061A.png)
5
6**Note:** The version of the ExternalTestRunner activity in this post does not handle tests based on Metadata parameters (blue box above), but should be OK for all other usages (it is just that these parameters have not been wired through yet). The red box show the new activity in place (this is the path taken if the tests are controlled by a test setting file) and the green box contains an MSTest activity waiting to be swapped out (this is the path taken if no test setting or metadata files are provided).
7
8The parameters on the activity in the red box are as follows, as said before they are basically the same as parameters for the standard MSTest activity.
9
10[](/wp-content/uploads/sites/2/historic/image_1009BCE5.png)
11
12The Result parameter (the Execute() method return value) does need to be associated with a variable declared in the workflow, in my case **ExternalTestRunnerResult**. This is defined at the sequence scope, the scope it is defined at must be such that it can be read by any other steps in the workflow that require the value. It is declared as being of the enum type **ExternalTestRunnerReturnCode** defined in the custom activity.
13
14[](/wp-content/uploads/sites/2/historic/image_36D7D325.png)
15
16Further on in the workflow you need to edit the if statement that branches on whether the tests passed or not to use this **ExtenalTestRunnerResult** value
17
18[](/wp-content/uploads/sites/2/historic/image_44AA1920.png)
19
20Once all this is done you should have all your MSTests running inside a Typemock’ed wrapper and all the results should be shown correctly in the build summary
21
22[](/wp-content/uploads/sites/2/historic/image_0F9D89F0.png)
23
24And the log of the build should show you all the parameters that got passed through to the MSTest program.
25
26[](/wp-content/uploads/sites/2/historic/image_366BA030.png)
27
28### Is there a better way to test a custom activity project?
29
30Whilst sorting out the logic for the custom activity I did not want to have to go through the whole process of running the team build to test the activity, it just took too long. To speed this process I did the following
31
321. In my solution I created a new Console Workflow project
332. I referenced my custom activity project from this new workflow project
343. I added my custom activity as the only item in my workflow
354. For each parameter of the custom activity I created a matching argument for the workflow and wired the two together.
36
37 [](/wp-content/uploads/sites/2/historic/image_763586B5.png)
38
395. I then created a Test Project that referenced the workflow project and custom activity project.
406. In this I could write unit tests (well more integration tests really) that exercise many of the options in the custom activity. To help in this process I created some simple Test Projects assemblies that contained just passing tests, just failing test and a mixture of both.
417. A sample test is shown below
42
43 ```
44 \[TestMethod\]
45 ``````
46 public void RunTestWithTwoNamedAssembly\_OnePassingOneFailingTestsNoPublishNoMstestSpecified\_FailMessage()
47 ``````
48 {
49 ```
50
51 ```
52 // make sure we have no results file, MSTest fails if the file is present
53 ``````
54 File.Delete(Directory.GetCurrentDirectory() + @"TestResult.trx");
55 ```
56
57 ```
58 var wf = new Workflow1();
59 ```
60
61 ```
62 Dictionary<string, object\> wfParams = new Dictionary<string, object\>
63 ``````
64 {
65 ``````
66 { "BuildNumber", string.Empty },
67 ``````
68 { "Flavour", "Debug" },
69 ``````
70 { "MsTestExecutable", string.Empty },
71 ``````
72 { "Platform", "Any CPU" },
73 ``````
74 { "ProjectCollection",string.Empty },
75 ``````
76 { "TeamProjectName", string.Empty },
77 ``````
78 { "TestAssemblyNames", new string\[\] {
79 ``````
80 Directory.GetCurrentDirectory() + @"TestProjectWithPassingTest.dll",
81 ``````
82 Directory.GetCurrentDirectory() + @"TestProjectWithfailingTest.dll"
83 ``````
84 }},
85 ``````
86 { "TestRunnerExecutable", @"C:Program Files (x86)TypemockIsolator6.0TMockRunner.exe" },
87 ``````
88 { "ResultsFile", "TestResult.trx" }
89 ``````
90 };
91 ```
92
93 ```
94 var results = WorkflowInvoker.Invoke(wf, wfParams);
95 ```
96
97 ```
98 Assert.AreEqual(TypemockBuildActivity.ExternalTestRunnerReturnCode.Failed, results\["ResultSummary"\]);
99 ``````
100 }
101 ```
102
1038. The only real limit here is that some of the options (the publishing ones) need a TFS server to be tested. You have to make a choice as to whether this type of publishing test is worth the effort of filling your local TFS server with test runs from the test project or whether you want to test these features manually in a real build environment, especially give the issues I mention in my past post
104
105### Summary
106
107So I have a working implementation of a custom activity that makes it easy to run Typemock based tests without losing any of the other features of a Team Build. Butt as I learnt getting around the deployment issues can be a real pain.