Running Test Suites within a network Isolated Lab Management environment when using TFS vNext build and release tooling

Sep 27, 2016 · 6 min read · Uncategorised ·

Share on:

Updated 27 Sep 2016: Added solutions to known issues

Background

As I have posted many times we make use of TFS Lab Management to provide network isolated dev/test environments. Going forward I see us moving to Azure Dev Labs and/or Azure Stack with ARM templates, but that isn’t going to help me today, especially when I have already made the investment in setting up a Lab Management environments and they are ready to use.

One change we are making now is a move from the old TFS Release Management (2013 generation) to the new VSTS and TFS 2015.2 vNext Release tools. This means I need to be able to trigger automated tests on VMs within Lab Management network isolated environments with a command inside my new build/release process. I have posted on how to do this with the older generation Release Management tools, turns out it is in some ways a little simpler with the newer tooling, no need to fiddle with shadow accounts etal.

My Setup

Constraints

The constraints are these

I need to be able to trigger tests on the Client VM in the network isolated lab environment. These tests are all defined in automated test suites within Microsoft Test Manager.
The network isolated lab already has a TFS Test Agent deployed on all the VMs in the environment linked back to the TFS Test Controller on my corporate domain, these agents are automatically installed and managed, and are handling the ‘magic’ for the network isolation – we can’t fiddle with these without breaking the Labs
The new build/release tools assume that you will auto deploy a 2015 generation Test Agent via a build task as part of the build/release process. This is a new test agent install, so removed any already installed Test Agent – we don’t want this as it breaks the existing agent/network isolation.
So my only options to trigger the tests by using TCM (as we did in the past) from some machine in the system. In the past (with the old tools) this had to be within the isolated network environment due to the limitation put in place by the use of shadow accounts.
However, TCM (as shipped with VS 2015) does not ‘understand’ vNext builds, so it can’t seem to find them by definition name/number – we have to find builds by their drop location, and I think this needs to be a UNC share, not a drop back onto the TFS server. So using TCM.EXE (and any wrapper scripts) probably is not going to deliver what I want i.e. the test run associated with a vNext build and/or release.

My Solution

The solution I adopted was to write a PowerShell script that performs the same function as the TCMEXEC.PS1 script that used to be run within the network isolated Labe Environment by the older Release Management products.

The difference is the old script shelled out to run TCM.EXE, my new version makes calls to the new TFS REST API (and unfortunately also to the older C# API as some features notably those for Lab Management services are not exposed via REST). This script can be run from anywhere, I chose to run it on the TFS vNext build agent, as this is easiest and this machine already had Visual Studio installed so had the TFS C# API available.

You can find this script on my VSTSPowerShell GitHub Repo.

The usage of the script is

 1TCMReplacement.ps1  
 2      -Collectionuri [http://tfsserver.domain.com:8080/tfs/defaultcollection/](http://tfsserver.domain.com:8080/tfs/defaultcollection/)   
 3      -Teamproject "My Project"  
 4      -testplanname "My test plan"    
 5      -testsuitename "Automated tests"   
 6      -configurationname "Windows 8"   
 7      -buildid  12345  
 8   -environmentName "Lab V.2.0"    
 9      -testsettingsname "Test Setting"   
10      -testrunname "Smoke Tests"   
11      -testcontroller "mytestcontroller.domain.com"  
12      -releaseUri "vstfs:///ReleaseManagement/Release/167"   
13      -releaseenvironmenturi "vstfs:///ReleaseManagement/Environment/247"

Note

The last two parameters are optional, all the others are required. If the last two are not used the test results will not be associated with a release
The is also a pollinginterval parameter which default to 10 seconds. The script starts a test run then polls on this interval to see if it has completed.
If there are any failed test then the script writes to write-error as the TFS build process sees this is a failed step

In some ways I think this script is an improvement over the TCMEXEC script, the old one needed you to know the IDs for many of the settings (loads of poking around in Microsoft Test Manager to find them), I allow the common names of settings to be passed in which I then use to lookup the required values via the APIs (this is where I needed to use the older C# API as I could not find a way to get the Configuration ID, Environment ID or Test Settings ID via REST).

There is nothing stopping you running this script from the command line, but I think it is more likely to make it part of release pipeline using the PowerShell on local machine task in the build system. When used this way you can get many of the parameters from environment variables. So the command arguments become something like the following (and of course you can make all the string values build variables too if you want)

 1   -Collectionuri $(SYSTEM.TEAMFOUNDATIONCOLLECTIONURI)   
 2   -Teamproject $(SYSTEM.TEAMPROJECT)   
 3   -testplanname "My test plan"  
 4   -testsuitename "Automated tests"  
 5   -configurationname "Windows 8"   
 6   -buildid  $(BUILD.BUILDID)  
 7  -environmentName "Lab V.2.0"  
 8   -testsettingsname "Test Settings"  
 9   -testrunname "Smoke Tests"   
10   -testcontroller "mytestcontroller.domain.com"   
11   -releaseUri $(RELEASE.RELEASEURI)   
12   -releaseenvironmenturi $(RELEASE.ENVIRONMENTURI)

Obviously this script is potentially a good candidate for a TFS build/release task, but as per my usual practice I will make sure I am happy with it’s operation before wrappering it up into an extension.

Known Issues

If you run the script from the command line targeting a completed build and release the tests run and are shown in the release report as well as on the test tab as we would expect.
However, if you trigger the test run from within a release pipeline, the test runs OK and you can see the results in the test tab (and MTM), but they are not associated within the release. My guess is because the release had not completed when the data update is made. I am investigating this to try to address the issue.

Previously I reported there was a known issue that the test results were associated with the build, but not the release. It turns out this was due to the AD account the build/release agent was running as was missing rights on the TFS server. To fix the problem I made sure the account as configured as follows”":

As a basic (or advanced) user who can access the test hub (http://:8080/tfs/_admin/_licenses) i.e. not a stakeholder
Also made sure the user was a ‘release administrator’ in the team project (http://:8080/tfs///_admin/_security/?_a=members)

Once this was done all the test results appeared where they should

So hopefully you will find this a useful tool if you are using network isolated environments and TFS build