But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Errors running tests via TCM as part of a Release Management pipeline

Whilst getting integration tests running as part of a Release Management  pipeline within Lab Management I hit a problem that TCM triggered tests failed as the tool claimed it could not access the TFS build drops location, and that no .TRX (test results) were being produced. This was strange as it used to work (the RM system had worked when it was 2013.2, seems to have started to be issue with 2013.3 and 2013.4, but this might be a coincidence)

The issue was two fold..

Permissions/Path Problems accessing the build drops location

The build drops location passed is passed into the component using the argument $(PackageLocation). This is pulled from the component properties, it is the TFS provided build drop with a \ appended on the end.

image 

Note that the \ in the text box is there as the textbox cannot be empty. It tells the component to uses the root of the drops location. This is the issue, as when you are in a network isolated environment and had to use NET USE to authenticate with a the TFS drops share the trailing \ causes a permissions error (might occur in other scenarios too I have not tested it).

Removing the slash or adding a . (period) after the \ fixes the path issue, so..

  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779        -  works
  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779\      - fails 
  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779\.     - works 

So the answer is add a . (period) in the pipeline workflow component so the build location is $(PackageLocation). as opposed to $(PackageLocation) or to edit the PS1 file that is run to do some validation to strip out any trailing characters. I chose the later, making the edit

if ([string]::IsNullOrEmpty($BuildDirectory))
    {
        $buildDirectoryParameter = [string]::Empty
    } else
    {
        # make sure we remove any trailing slashes as the cause permission issues
        $BuildDirectory = $BuildDirectory.Trim()
        while ($BuildDirectory.EndsWith("\"))
        {
            $BuildDirectory = $BuildDirectory.Substring(0,$BuildDirectory.Length-1)
        }
        $buildDirectoryParameter = "/builddir:""$BuildDirectory"""
    }
   

Cannot find the TRX file even though it is present

Once the tests were running I still had an issue that even though TCM had run the tests, produced a .TRX file and published it’s contents back to TFS, the script claimed the file did not exist and so could not pass the test results back to Release Management.

The issue was the call being used to check for the file existence.

[System.IO.File]::Exists($testRunResultsTrxFileName)

As soon as I swapped to the recommended PowerShell way to check for files

Test-Path($testRunResultsTrxFileName)

it all worked.

‘Test run must be created with at least one test case’ error when using TCM

I have been setting up some integration tests as part of a release pipeline. I am using TCM.EXE to trigger tests from the command line. Something along the lines

TCM.exe run /create /title:"EventTests" /collection:"http://myserver:8080/tfs” /teamproject:myteamproject /testenvironment:"Integration" /builddir:\\server\Drops\Build_1.0.226.1975”  /include /planid:26989  /suiteid:27190 /configid:1

I kept getting the error

‘A test run must be created with at least one test case’

Strange thing was my test suite did contains a number of test, and they were marked as active.

The issue was actually the configid it was wrong, there is no easy way to check them from the UI. use the following command to get a list of valid IDs

TCM.exe configs /list   /collection:"http://myserver:8080/tfs” /teamproject:myteamproject

Id        Name
--------- ----------------------------------------------------------------
35        Windows 8.1 ARM
36        Windows 8.1 64bit
37        Windows 8.1 ATOM
38        Default configuration created @ 11/03/2014 12:58:15
39        Windows Phone 8.1

Your can now use the correct ID, not one you had to guess

Review of ‘Software Testing using Visual Studio 2012’ from Packt Publishing

I have just been reading Software Testing using Visual Studio 2012 by Subashni. S and Satheesh Kumar. N from Packt Publishing

9540EN_cov-stwvs2012

This book does what it says on the cover, it is a general introduction to the testing tools within the Visual Studio 2012 family. My comment is not about how well it is done, it is a clear enough introduction, but why produce a book that really just covers what is in MSDN, Channel9, numerous podcasts, blogs and ALM Rangers documentation?

I suppose this is a question of target audience, some people like to browse a physical book for ‘new’ technology, I can see that (though I tried it on Kindle, more of that later). This book certainly does cover the core areas, but sits strangely between a technology briefing for a manager/person who just needs an overview (it is all a bit long winded, list all the features and flags of tools) and not enough detail for the practitioner (the exercises do not go deep enough unlike those provide by Microsoft in Brian Keller VS/TFS demo VM series)

Given this concern I wonder who the target audience really is?

A real issue here is that Microsoft have gone to quarterly updates, so the product is always advancing, faster than any print book can manage (Microsoft’s own MSDN documentation has enough problems keeping up, and frequently is play catch up). For a book on testing this is a major problem as ‘test’ has been a key focus for the updates. This means when the book’s contents is compared to Visual Studio/TFS 2012.3 (the current shipping version at the time of this review) there are major features missing such as

  • The improvements in Test Explorer to support other non Microsoft test framework, playlists etc.,
  • SKU changes in licensing, MTM dropping down to Premium form Ultimate
  • Azure based load testing
  • The test experience in the web browser (as opposed to MTM)

The list will always grow while Microsoft stick to their newer faster release cycle. This was not too much of a problem when Microsoft shipped every couple of years, a new book opportunity, but now how can any book try to keep up on a 12 week cycle?

One option you would think is Kindle or eBooks in general, as at least the book can be updated . However there is still the issue of the extra effort of the authors and editors, so in general I find these updates are not that common. The authors will usually have moved onto their next project and not be focused on yet another unpaid update to a book they published last quarter.

As to my experience on the Kindle, this was the first technical book I have read on one. I have used the Kindle App on a phone for a couple of years for my novel reading, but always felt the screen was too small for anything that might have a diagram in it. I recently bought a Kindle Paperwhite so though I would give this book a go on it. I initially tried to email the book from the Packt site straight to my Kindle, but this failed (a file size issue I am told by Packt customer support), but a local copy of USB was fine.

So how was the Kindle experience? OK, it did the job, everything was clear enough,  it was not a super engaging reading experience but it is a technical book, what do you expect? It was good enough that I certainly don’t see my getting too many paper books going forward whether thet be novels or technical books.

So in summary, was the book worth the effort to read? I always gauge this question on ‘did I learn something?’ and I did. There is always a nugget or two in books on subjects you think you know. However, ‘would I say it is a really useful/essential read for anyone who already has a working knowledge in this subject?’, probably not. I would say their time is better spent doing a hand on lab or watching conference recordings on Channel9.

Leave this book to anyone who wants a general written introduction to the subject of Microsoft specific testing tooling.

TFS Test Agent cannot connect to Test Controller – Part 2

I posted last week on the problems I had had getting the test agents and controller in a TFS2012 Standard environment talking to each other and a workaround. Well after a good few email with various people at Microsoft and other consultants at Black Marble I have a whole range of workarounds solutions.

First a reminder of my architecture, and note that this could be part of the problem, it is all running on a single Hyper-V host. Remember this is a demo rig to show the features of Standard Environments. I think it is unlikely that this problem will be seen in a more ‘realistic’ environment i.e. running on multiple boxes

 

image

 

The problem is that when the test agent running on the Server2008 should request the test controller (running the on VSTFS server) should call it back on either it 169.254.x.x address or on abn address obtained via DHCP from the external virtual switch. However the problem is it is requesting a call back on 127.0.0.1, as can be seen in the error log

Unable to connect to the controller on 'vstfs:6901'. The agent can connect to the controller but the controller cannot connect to the agent because of following reason: No connection could be made because the target machine actively refused it 127.0.0.1:6910. Make sure that the firewall on the test agent machine is not blocking the connection.

The root cause

It turns out the root cause of this problem was I had edited the c:\windows\system32\drivers\etc\hosts file on the test server VM to add an entry to allow a URL used in CodedUI tests to be resolved to the localhost

127.0.0.1   www.mytestsite.com

Solution 1 – Edit the test agent config to bind to a specific address

The first solution is the one I outlined in my previous post, tell the test agent to bind to a specific IP address. Edit

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\QTAgentService.exe.config

and added a BindTo line with the correct address for the controller to call back to the agent

<appSettings>
     // other bits …
      <add key="BindTo" value="169.254.1.1"/>
</appSettings>

The problem with this solution you need to remember to edit a config file, all seems a bit complex!

Solution 2 – Don’t resolve the test URL to localhost

Change the hosts file entry used by the CodedUI test to resolve to the actual address of the test VM e.g.

169.254.1.1   www.mytestsite.com

Downside here is you need to know the test agents IP address, which depending on the system in use could change, and will certainly be different on each test VM in an environment. Again all seems a bit complex and prone to human error.

Solution 3 – Add an actual loopback entry to the hosts file.

The simplest workaround which Robert Hancock at Black Marble came up with was to add a second entry to the hosts file for the name loopback

127.0.0.1   localhost
127.0.0.1   www.mytestsite.com

Once this was done the test agent could connect, I did not have to edit any agent config files, or know the address the agent need to bind to. By far the best solution

 

So thanks to all who helped get to the bottom of this surprisingly complex issue.

Experiences upgrading an MVC 1 application to MVC 3

I have recently had to do some work on a MVC 1 application and thought it sensible to bring it up to MVC 3, you don’t want to be left behind if you can avoid it. This was a simple data capture application written in MVC1 in Visual Studio 2008 and never needed to be touched since. A user fills in a form, the controller then takes the form contents and stores it. The key point to note here is that it was using the Controller.UpdateModel<TModel> Method (TModel, IValueProvider) method, so most of the controller actions look like

[AcceptVerbs(HttpVerbs.Post)]
       public ActionResult PostDataToApplication(FormCollection form)
       {
           NewApplication data = new NewApplication();
           try
           {
               UpdateModel(data, form.ToValueProvider());
              
               // process data object
               bool success = DoSomething(data);

               if (success)
               {
                   return RedirectToAction("FormSuccess", "Home");
               }
               else
               {
                   return RedirectToAction("FormUnsuccessful", "Home");
               }
           }
           catch (InvalidOperationException)
           {
               return View();
           }
       }

This worked fine on MVC 1 on VS2008, but I wanted to move it onto MVC3 on VS2012 if possible; with a little changes as possible as this is a small web site that has very few changes so not worth a major investment in time to keep updated to current frameworks. So these were steps I took and gotcha’s I found

The upgrade itself

First I opened the VS2008 solution in VS2010 and it automatically upgraded to MVC2, a good start!

I then used the MVC2 to MVC3 tool on Codeplex, this initially failed and it took me a while to spot that you can only use this tool if your MVC2 application targets .NET 4. Once I changed the MVC2 to target .NET 4 as opposed to 3.5 this upgrade tool worked fine.

I could now load my MVC3 application in either VS2010 or VS2012.

Using the Web Site

At this point I though I had better test it, and instantly saw a problem. Pages that did not submit data worked fine, but submitting a data capture forms failed with Null Exception errors. Turns out the problem was a change in default behaviour of the models between MVC releases. On MVC1 empty fields on the form were passed as empty strings, with MVC 2(?) and later they are passed as nulls.

Luckily the fix was simple. Previously my model had been

public class SomeModel: IDataErrorInfo
{
     public string AgentNumber { get; set; }
     …..
}

I needed to add a  [DisplayFormat(ConvertEmptyStringToNull = false)] attribute on  each string property to get back to the previous behaviour my controller expected

public class SomeModel: IDataErrorInfo
{
    [DisplayFormat(ConvertEmptyStringToNull = false)]
     public string AgentNumber { get; set; }
     …..
}

Now my web site ran as I had expected.

Unit Tests

I had previously noticed my unit tests were failing. I had expected the change to the model would fix this too, but it did not. On the web there a good many posts as to how unit testing of MVC2 and later fails unless you mock out the controller.context. You see errors in the form

Test method Website.Tests.Controllers.HomeControllerTest.DetailsValidation_AlphaInValidAccountID_ErrorMessage threw exception:
System.ArgumentNullException: Value cannot be null.
Parameter name: controllerContext
Result StackTrace:   
at System.Web.Mvc.ModelValidator..ctor(ModelMetadata metadata, ControllerContext controllerContext)
   at System.Web.Mvc.ModelValidator.CompositeModelValidator..ctor(ModelMetadata metadata, ControllerContext controllerContext)
   at System.Web.Mvc.ModelValidator.GetModelValidator(ModelMetadata metadata, ControllerContext context)
   at System.Web.Mvc.DefaultModelBinder.OnModelUpdated(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.DefaultModelBinder.BindComplexElementalModel(ControllerContext controllerContext, ModelBindingContext bindingContext, Object model)
   at System.Web.Mvc.DefaultModelBinder.BindComplexModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.DefaultModelBinder.BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.Controller.TryUpdateModel[TModel](TModel model, String prefix, String[] includeProperties, String[] excludeProperties, IValueProvider valueProvider)
   at System.Web.Mvc.Controller.UpdateModel[TModel](TModel model, String prefix, String[] includeProperties, String[] excludeProperties, IValueProvider valueProvider)
   at System.Web.Mvc.Controller.UpdateModel[TModel](TModel model, IValueProvider valueProvider)
   at Website.Controllers.HomeController.Details(FormCollection form)

The fix is to not just new up a controller in your unit tests like this

HomeController controller = new HomeController();

But to have a helper method to mock it all out (which is created for MVC associated test projects for you, so it is easy)

private static HomeController GetHomeController()
{
   IFormsAuthentication formsAuth = new MockFormsAuthenticationService();     
   MembershipProvider membershipProvider = new MockMembershipProvider();
   RoleProvider roleProvider = new MockRoleProvider();

   AccountMembershipService membershipService = new AccountMembershipService(membershipProvider, roleProvider);
   HomeController controller = new HomeController(formsAuth, membershipService);
   MockHttpContext mockHttpContext = new MockHttpContext();

   ControllerContext controllerContext = new ControllerContext(mockHttpContext, new RouteData(), controller);
   controller.ControllerContext = controllerContext;
   return controller;
}

However, this problem with a missing context was not my problem, I was already doing this. The error my test runner was showing did not mention the context, rather binding errors.

Test method CollectorWebsite.Tests.Controllers.HomeControllerTest.CardValidation_AlphaInValidAccountID_ErrorMessage threw exception:
System.NullReferenceException: Object reference not set to an instance of an object.
Result StackTrace:   
at CollectorWebsite.Models.CardRecovery.get_Item(String columnName) 
   at System.Web.Mvc.DataErrorInfoModelValidatorProvider.DataErrorInfoPropertyModelValidator.Validate(Object container)
   at System.Web.Mvc.ModelValidator.CompositeModelValidator.<Validate>d__5.MoveNext()
   at System.Web.Mvc.DefaultModelBinder.OnModelUpdated(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.DefaultModelBinder.BindComplexElementalModel(ControllerContext controllerContext, ModelBindingContext bindingContext, Object model)
   at System.Web.Mvc.DefaultModelBinder.BindComplexModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.DefaultModelBinder.BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
   at System.Web.Mvc.Controller.TryUpdateModel[TModel](TModel model, String prefix, String[] includeProperties, String[] excludeProperties, IValueProvider valueProvider)
   at System.Web.Mvc.Controller.UpdateModel[TModel](TModel model, String prefix, String[] includeProperties, String[] excludeProperties, IValueProvider valueProvider)
   at System.Web.Mvc.Controller.UpdateModel[TModel](TModel model, IValueProvider valueProvider)
   at CollectorWebsite.Controllers.HomeController.CardRecovery(FormCollection form)

I got stuck here for a good while………

Then it occurred to me if the behaviour has changed such that on the web site I see nulls when I expect empty strings, I bet the same is happening in unit tests. It is trying to iterate though what was a collection of strings and is now at best a collection of nulls or just an empty collection. The bind failed as it could not match the form to the data.

The fix was to make sure in my unit tests I passed in a FormCollection that had all the expected fields (with suitable empty values e.g string.empty). This meant my unit tests changed from

[TestMethod, Isolated]
public void ApplicationValidation_CurrentPostcodeNumbersOnly_ErrorMessage()
{

           // Arrange
           HomeController controller = GetHomeController();
      
    FormCollection form = new FormCollection();
           form.Add("CurrentPostcode", "12345");

           // Act
           ViewResult result = controller.Application(form) as ViewResult;

           // Assert
           Assert.IsNotNull(result);
           Assert.AreEqual("Please provide a valid postcode", result.ViewData.ModelState["CurrentPostcode"].Errors[0].ErrorMessage);

       }

To

[TestMethod, Isolated]
public void ApplicationValidation_CurrentPostcodeNumbersOnly_ErrorMessage()
{

           // Arrange
           HomeController controller = GetHomeController();
          
FormCollection form = GetEmptyApplicationFormCollection();
           form.Set("CurrentPostcode", "12345");

           // Act
           ViewResult result = controller.Application(form) as ViewResult;

           // Assert
           Assert.IsNotNull(result);
           Assert.AreEqual("Please provide a valid postcode", result.ViewData.ModelState["CurrentPostcode"].Errors[0].ErrorMessage);

       }

where the GetEmptyApplicationFormCollection() helper method just creates a FormCollection with all the forms fields.

Once this was done my unit test passed.

Summary

So I now have an MVC3 application that works and passes unit tests. You could argue I should do more work so it does not need these special fixes, but it meets my needs for now.

Why Typemock Isolator does not work on TFS 2012 Build and what you can do about it

Update 4/Aug/2012: See post on my implementation of a 2012 version of this activity

If you are using Typemock Isolator in your unit testing then you be wanting to include then in your automated build process. Unlike other mocking frameworks you have to do a bit of work to achieve this with Isolator, this is because to enable its advanced features of mocking items like sealed private classes you have to start the interceptor that does the magic prior to running the tests and the switch it off afterwards.

In the past I posted on how you could use a build activity I wrote to wrapper MStest in a TFS 2010 build using TMockRunner. Since I wrote this Typemock released a set of build activities to do the starting and stopping of the inceptor as separate activities, a much more flexible solution which I would always recommend for TFS.

However when you try to use either with TFS 2012 you get a problem, the activities fail to load. This is a problem we saw on the TFS Extensions Codeplex project; you have to build you activities against either the 2010 TFS API or the 2012 TFS API. You don’t need to alter the code in your activities, but you do need to make a specific build.

So at this time there is no solution, one or both of these activities need to be rebuilt. For the MSTest wrapper I wrote the source is available so you can do it yourself if you want to, but the way Typemock have implemented their activities is a better solution. This is because it is not reliant on TMockRunner and MStest, it can wrapper any activities. This is important as to be able to use the new ‘any framework’ unit testing features in VS2012 you want to use Microsoft’s new test running activities and not just the old MSTest activity.

I understand that Typemock are looking at releasing a TFS2010 version of their activities soon, but I know of no release date as yet. If you want an immediate solution you will need to do a bit of work.

  • You could rebuild my MSTest based activity
  • You could use the standard InvokeMethod activity and put the contents of my MStest activity’s generated command line into this
  • But the one I favour is to use a dissembler such as Telerik JustDecompile to get the code from the Typemock.TFS2010.DLL and build a new activity.

However, it must be said I see this as just a temporary measure until the official Typemock 2012 activity is released. I am not sure I will get around to doing this before the Typemock release, we shall see.

DDD South West session on Unit testing in VS11

Thanks to everyone who attended my session at DDDSW today. The session was completely demo driven so no slides to share, but the contents of the session is covered in the blog posts