Just had a problem with a TFS 2012 Lab Management deployment build. It was working this morning, deploying two web sites via MSDeploy and a DB via a DacPac, then running some CodedUI tests. However, when I tried a new deployment this afternoon it kept failing with the error:
The deployment task was aborted because there was a connection failure between the test controller and the test agent.
If you watched the build deployment via MTM you could see it start OK, then the agent went off line after a few seconds.
Turns out the solution was the old favourite, to reboot of the Build Controller. Would like to know why it was giving this intermittent problem though.
Update 14th Jan An alternative solution to rebooting is to add a hosts file entry on the VM running the test agent for the IP address of the test controller. Seems the problem is name resolution, but not sure why it occurs
Lab Management has a lot of moving parts, especially if you are using SCVMM based environments. All the parts have to communicate if the system is work.
One of the most common problem I have seen are due to DNS issues. A slowly propagating DNS can cause chaos as the test controller will not be able to resolve the name of the dynamically registered lab VMs.
The best fix is to sort out your DNS issues, but that is not always possible (some things just take the time they take, especially on large WANs).
An immediate fix is to use the local host files on the test controller to define IP address for the lab[guid].corp.domain names created when using network isolation. Once this is done the handshake between the controller and agent is usually possible.
If it isn’t then you are back to all the usually diagnostics tools
We have recently gone through an exercise to provide a consistent set of prebuilt configured VMs for use in our TFS Lab Management environments.
This is not an insignificant piece of work as this post by Rik Hepworth discusses detailing all the IT pro work he had to do to create them. This is all before we even think about the work required to create deployment TFS builds and the like.
It is well worth a read if you are planning to provision a library of VM for Lab Management as it has some really useful tips and tricks
I was working on a TFS lab deployment today and to speed up testing I set it to pick the <Latest> build of the TFS build that actually builds the code, as opposed to queuing a new one each time. It took me ages to remember/realise why it kept trying to deploy some ancient build that had long since been deleted from my test system.
The reason was the <Latest> build means the last successful build, not the last partially successful build. I had a problem with my code build that means a test was failing (a custom build activity on my build agent was the root cause if the issue). Once I fixed my build box so the code build did not fail the lab build was able to pickup the files I expected and deploy them
Note that if you tell the lab deploy build to queue it’s own build it all attempt to deploy this even if it is partially successful.
I recently posted about adding extra VMs to existing environments in Lab Management. Whilst following this process I hit a problem, I could not create my new environment there was a problem at the verify stage. It was fine adding the new VMs, but the one I was reusing gave the error ‘Microsoft test manager cannot install test agent on these machines because another environment is being created using the same machines’
II had seen this issue before and so I tried a variety of things that had sorted it in the past, removing the TFS Agent on the VM, manually installing and trying to configure them, reading through the Test Controller logs, all to no effect. I eventually got a solution today with the help of Microsoft.
The answer was to do the following on the VM showing the problem
- Kill TestAgentInstaller.exe process, if running on failing machine
- Delete “TestAgentInstaller” service from services, using sc delete testagentinstaller command (gotcha here, use a DOS style command prompt not PowerShell as sc has a different default meaning in PowerShell, it is an alias for set-content. if using PowerShell you need the full path to the sc.exe)
- Delete c:\Windows\VSTLM_RES folder
- Restart machine and then try Lab Environment creation again and all should be OK
- As usual once the environment is created you might need to restart it to get all the test agents to link up to the controller OK
So it seems that the removal of the VM from its old environment left some debris that was confusing the verify step. Seems this only happens rarely but can be a bit of a show stopper if you can’t get around it
If you are using network isolated environment in TFS Lab management there is no way to add another VM unless you rebuild and redeploy the environment. However, if you are not network isolated you can at least avoid the redeploy issues to a degree.
I had a SCVMM based environment that was a not network isolated environment that contained a single non-domain joined server. This was used to host a backend simulation service for a project. In the next phase of the project we need to test accessing this service via RDP/Terminal Server so I wanted to add a VM to act in this role to the environment.
So firstly I deleted the environment in MTM, as the VMs in the environment are not network isolated they are not removed. The only change is to remove the XML meta data from the properties description.
I now needed to create my new VM. I had thought I could create a new environment adding the existing deployed and running VM as well as a new one from the SCVMM library. However you get the error ‘ cannot create an environment consisting of both running and stored VMs’
So here you have two options.
- Store the running VM in the library and redeploy
- Deploy out, via SCVMM, a new VM from some template or stored VM
Once this is done you can create the new environment using the running VMs or stored images depending on the option chosen in the previous step.
So not any huge saving in time or effort. Just wish there was a way to edit deployed environments
An interesting change with Lab Management 2012 and SCVMM 2012 is that templates become a lot less useful. In the SCVMM 2008 versions you had a choice when you stored VMs in the SCVMM library. …
- You could store a fully configured VM
- or a generalised template.
When you added the template to a new environment you could enter details such as the machine name, domain to join and product key etc. If you try this with SCVMM 2012 you just see the message ‘These properties cannot be edited from Microsoft Test Manager’
So you are meant to use SCVMM to manage everything about the templates, not great if you want to do everything from MTM. However, is that the only solution?
An alternative is to store a SYSPREP’d VM as a Virtual Machine in the SCVMM library. This VM can be added as many times as is required to an environment (though if added more than once you are asked if you are sure)
This method does however bring problems of its own. When the environment is started, assuming it is network isolated, the second network adaptor is added as expected. However, as there is no agent on the VM it cannot be configured, usually for a template Lab Management would sort all this out, but because the VM is SYSPREP’d it is left sitting at the mini setup ‘Pick your region’ screen.
You need to manually configure the VM. So the best process I have found is
- Create the environment with you standard VMs and the SYSPRED’d one
- Boot the environment, the standard ready to use VMs get configured OK
- Manually connect to the SYSPREP’d VM and complete the mini setup. You will now have a PC on a workgroup
- The PC will have two network adapters, neither connected to you corporate network, both are connected to the network isolated virtual LAN. You have a choice
Either way you need to manually install the Test Agent and run the configuration (just select the defaults it should know where the test controller is). This will configure network isolated adaptor to the 192.168.23.x network Now you can manually join the isolated domain A reboot the VM (or the environment) and all should be OK
- Connect the legacy adaptor to your corporate LAN, to get at a network share via SCVMM
- Mount the TFS Test Agent ISO
All a bit long winded, but does mean it is easier to build generalised VMs from MTM without having to play around in SCVMM too much.
I think all would be a good deal easier of the VM had the agents on it before the SYSPREP, I have not tried this yet, but that is true in my option of all VMs used for Lab Management. Get the agents on early as you can, just speeds everything up.
Thanks to everyone who attended my webinar session today in TFS Lab Management, I hope the audio issues were not too much of a problem and you found the session useful. We are looking into the causes of the audio problem so hopefully the next webinar will not need people dialling in via the phone unless they want to.
A number of people asked for the slides, you can find a copy of them here.
As I mentioned in the session if you want a look at Lab Management you can have a go yourself using the HOL in Brian Keller’s TFS 2012 VM. Or watch the video I did at Techday 2010, an end to end demo of Lab Management.
I also mentioned a couple of Microsoft case studies that might be of interest
If you are interested finding out more about TFS Lab Management why not come to the Black Marble Bite size webinar next Tuesday (23rd April). I will be giving a basic overview of the product and discussing some of our experiences implementing it.