But it works on my PC!

We have been building ‘standard’ environments for our TFS Lab Management system. Environments that can be used for most of the projects we are involved in without too much extra setup e.g. a small domain controller VM and a Server VM with SQL and SharePoint. These environments have a series of snapshots so it can be used in a number of ways e.g if we just want SQL and IIS we just go back to a snapshot prior to SharePoint being installed.

When trying to deploy one of these environments we saw a couple issues.

Capacity

First we got the error that there was not a suitable host with enough capacity to host the environment (remember all the VMs in a network isolated environment need to be on the same Hyper-V host). This can be a bit of a red herring as with dynamic memory and other new Hyper-V features there is often the capacity there (see Tiago’s post on this for more details). The fix here was to set TFS to allow aggressive deployment using the command

C:\Program Files\Microsoft Team Foundation Server 2010\Tools>tfsconfig lab /hostgroup /collectionName:myTpc  ​/labenvironmentplacementpolicy:aggressive /edit /Name:"My hosts group"

Initial Startup

The next problem I saw was that when the new environment was deployed it did not started cleanly. The first time an environment is started it seems to take longer than subsequent starts (assume there is some initial configuration done). Basically in this case network isolation did not start correctly, hence build and testing capabilities also failed.

The fix was simple, shut down the environment and start it again. The tried and trusted IT answer to all problems. This time it started fine, and was faster to start.

Now I have not see this issue every time I deploy. When I deployed the same environment again and it worked perfectly first time. I suspect it was really a capacity issue on the underlying Hyper-V server causing some delay, but I am running in aggressive mode so I should expect this.

A common cause of the TF266026 error is because when the build agent tries to start (it is the build agent that runs the workflows in Lab Management) it cannot access the custom assemblies folder as defined for its parent build controller. Obviously this problem only occurs if you have  set a custom assemblies path for parent build controller.

The reason for the error is because the agent is running as the Lab Management service account, in my case tfs2010lab, as defined for the TPC in the TFS Administration Console. This account by default has no rights to the source folder assigned for the custom assemblies. This is not usually an issue until it needs to access source control to load custom assemblies (which actually it probably does not ever use as it is not building code!).

As soon as this service account is granted access  to this folder, by making it a  reader, contributor or builder on the team project, the problem goes away.

It can be a bit confusing to get work out which tools to use at which stages required to get a lab environment up and running in TFS. Here is a basic workflow showing what you need to do in System Center Virtual Machine Manager prior to starting in MTM Lab Center

Note: if you want to copy environments between TFS Team project Collections have a look at this post

I got a TF260073, incompatible architecture error when trying to deploy a new virtual lab environment using a newly created VM and template. I found the fix in a forum post.

The issue was that when I had build the VMs, I had installed the Lab Management agents using a VMprep DVD ISO and mounted it using ‘share image instead of copying it’ option. This as the name implies means the ISO is mount from a share not copied to the server running the VM, this save time and disk resources. When I had stored my VM into the SCVMM Library I had left this option selected i.e the VMPrep.iso mounted. All I had to do to fix this issue was open the settings of the VM stored in the SCVMM Library and dismount the ISO, as shown below

Interestingly the other VM I was using in my environment was stored as template and did not suffer this problem. When creating the template I was warning that it could not be created if an ISO was mounted in this manner. So the fact I had a problem with my VM image should not have been a surprise.

Background

One area I think can be found confusing in TFS Lab Management is that all environments are associated with specific Team Projects (TP) within Team Project Collections (TPC). This is not what you might first expect if you think of Lab Management as just a big Hyper-V server. When configured you end up with a number of TPC/TP related silos as shown in the diagram below.

This becomes a major issue for us as each TP stores its own environment definitions in its own silo; they cannot be shared between TPs and hence TPCs. So it is hard to re-use environments without recreating them.

This problem effects companies like ourselves as we have many TPCs because we tend to have one per client, an arrangement not that uncommon for consultancies.

It is not just in Lab Management this is an issue for us. The isolated nature of TPCs, a great advantage for client security, has caused us to have an ever growing number of Build Controllers and Test Controllers which were are regularly reassigning to whichever are our active TPCs. Luckily multiple the Build Controller can be run on the same VM (I discussed this unsupported hack here), but unfortunate there is no similar workaround for Test Controllers.

MTM is not your friend when  storing environments for use beyond the current TP

What I want to discuss in this post is how, when you have a working environment in one TP you can get it into another TP with as little fuss as possible.

Naively you would think that you use the Store in Library option within MTM that is available for a stopped environment.

This does store the environment on the SCVMM Library, but it is only available for the TP that is was stored from. It is stored in the A1 silo in the SCVMM Library. Now you might ask why, the SCVMM Library is just a share, so anything in it should be available to all? But it turns out it is not just a share. It is true the files are on a UNC share, you can see the stored environments as a number of Lab_[guid] folders, but there is also a DB that stores meta data, this is the problem. This meta data associates the stored environment with a given TP.

The same is true if you choose to just store a single VM from within MTM whether you choose to store it as a VM or a template.

Why is this important you might ask? Well it is all well and good you can build your environment from VMs and templates in the SCVMM Library, but these will not be fully configured for your needs. You will build the environment, making sure TFS agents are in place, maybe putting extra applications, tools or test data on system. It is all work you don’t want to have to repeat for what is in effect the same environment in another TP or TPC. This is a problem we see all the time. We do SharePoint development so want a standard environment (couple of load balanced servers and a client) we can use for many client projects in different TPCs  (Ok VM factory can help, but this is not my point here).

A workaround of sorts

The only way I have found to ease this problem is when I have a fully configured environment to clone the key VMs (the servers) into the SCVMM Library using SCVMM NOT MTM

1. Using MTM stop the environment you wish to work with.
2. Identify the VM you wish to store, you need its Lab name. This can be found in MTM if you connect to the lab and check the system info for the VM
   
   
3. Load SCVMM admin console, select Virtual Machines tab and find the correct VM
   
   
4. Right click on the VM and select Clone
5. Give the VM as new meaningful name e.g. ‘Fully configured SP2010 Server’
6. Accept the hardware configuration (unless you wish to change it for some reason)
7. IMPORTANT On the destination tab select the to ‘store the virtual machine in the library’. This appears to be the only means to get a VM into the library such that it can be imported into any TPC/TP.
   
   
8. Next select the library share to use
9. And let the wizard complete.
10. You should now have a VM in the SCVMM Library that can be imported into new environments.

You do have to at this point recreate the environment in your new TP, but at least the servers you import into this environment are configured OK. If for example you have a pair of SP2010 servers, a DC and a NLB, as long as you drop them into a new isolated environment they should just leap into life as they did before. You should not have to do any extra re-configuration.

The same technique could be used for workstation VMs, but it might be as quick to just use template (sysprep’d) clients. You just need to take a view on this for your environment requirements

Today we got Direct Access working; I can now access all the features of our TFS Lab Management instance from home. No more remoting onto on box to hop to another. I never need to go to the office again!

I did a post at the start of the year about Lab management and VLAN tags, how they are not supported, but you can work around the problems. Over the past few months we have split our old Hyper-V cluster into one for production and one for test/lab development. This gave our IT team a chance to look at the VLAN problem again.

So a quick reminder of the issue – the deployment tools in Lab management that create environments provide no means to set a VLAN tag for any networks connections they create. Once an environment is created you can manually set a VLAN tag, but it is all a bit of a pain and certainly unsupported.

The solution our IT team have come up with to avoid the problem is to set the default VLAN tag on the physical port on the Ethernet switch. Hence any VMs/Environments on the the new test/lab Hyper-V don’t have to worry about VLANs at all, they are all automatically, in our case, on subnet 200. This works for TFS Lab Management and also means our developers need to have no knowledge of IP routing setup to deploy a VM/environment. Our production Hyper-V box, that runs much of our business systems, still uses manually set VLAN tagging as before, but as there is no auto deployment involved on this system there are no problems.

There is one gotcha though…..

If you try to use a VM created on our old setup, that was previously set with the VLAN tag of 200, it cannot see the LAN, even though it has what you think is the correct VLAN tag. This is because setting a VLAN tag with Hyper-V to 200 is not the same as not setting a VLAN tag in the operating system and letting the Ethernet switch default the port to the VLAN tag 200. So you have to let the switch manage the VLAN tag, the VM needs to know nothing about it. As shown below

So once this is all set you have your routed network, but also have a fully supported Lab Management setup

When using VMPrep to setup a VM for use in a Lab Management system I got the error cross at the bottom of the dialog

Not too much help, usually there is a link to the log file or a message.

If you look in the log file in c:\user\[name]\Appdata\Roaming\LMInstaller.txt you see that the path to the Patches folder is invalid.

This is fixed by editing the VMPrepTool\VMPrepToolLibrary\Applications.XML file and correcting the path (which I had made a typo in)

I have just replaced my old Acer laptop with a rather nice Lenovo W520. This has plenty of memory and is able to run Hyper-V. In the past for TFS demos I had used boot from VHD to boot the Acer into Windows 2K8, as the Acer could not run Hyper-V due to lack of hardware virtualisation support in the bios. So I had a fully configured VHD boot what I wanted to move to Hyper-V.

My first though was I could just use the P2V system build into SCVMM. I ran the wizard, connected to my VHD booted Acer laptop, provide the details asked for and all looked good until the last screen when I got the error

Error (13256):
The disk with index 1 on source machine 192.168.100.30 is an attached virtual hard disk. This configuration is not supported for physical-to-virtual conversions.

So that was a non starter.

So I just copied the VHD to my new Hyper-V server and created a new Hyper-V VM using it. However, when I tried to boot this it said I had no boot sector. When you think about it this not unreasonable as this VHD was always booted off my laptops primary disk boot partition. The process to fix this was as follows (thanks to Rik for some help here)

1. Mount the VHD directly onto my Windows 2K8 Hyper-V host via the Disk Manager in the admin tools.Don’t bother to assign a drive letter.
2. Select the Windows 2K8 partition on the VHD and shrink it by 110Mb, this is to create space for the boot partition (I suppose you could use a VHD resizer tool, but that would be slower from my experience as this rewrites the whole VHD, the shrink is very quick)
3. In the new gap, create a simple partition and format as NTFS with the same ‘System Reserved’
4. Dismount the VHD
5. Start the Hyper-V VM using the edit VHD, you will still get the no boot device error
6. Attach a Windows 7 DVD ISO and restart the VM, it should boot into the Windows setup. On the first screen press Shift F10 to get the command prompt.
7. You should be on drive X: and see a drive C; (your old VHD partition) and D: ( the newly created one)
8. Run the command bcdboot c:\windows /s d: to create the boot partition
9. Load diskpart and (probably) the following commands
• select the VHD disk – sel disk 0
• list the partitions – list part
• select the new partition – sel part 2
• make it active – active
• exit distpart
10. As a check you can run the command bcdedit to see that it added something (this command would have returned nothing prior to bcdboot being run)

You should now be able to restart the VM and it should boot using the installed Windows 2K8 partition. As it has changed hardware it will probably want to reboot a few times as drivers are updated.

Whist on the road I have had need to access our Lab Management system via our TMG firewall through which we expose our TFS 2010 for remote users (via SSL). When I load Microsoft Tests Manager (MTM) I can connect to the TFS server, as expected, and go into ‘Lab Center’ mode. I can see my project’s environment and can start, stop and deploy them without issue (all communication routed via our TFS server). However the MTM environment viewer fails to make a connection to the test VMs in the environments.

MTM environment viewer can connect to an environment in two ways:

• Host connection – via the Hyper-V management protocols
• Guest connection – via a RDP session to the VM’s operating system

From outside our firewall a host connection is not an option as the required ports are not open. So my only option was a guest connection. However, our TMG firewall is set to provide a  RD gateway, effectively a proxy for RDP sessions. You have to configure RDP to use this, and have to authenticate with this gateway prior to authenticating with the actual target remote machine.

The problem is MTM does not support the use of TMG RD Gateways.

However there is a solution. If I right click on the VM in MTM Environment Viewer you can launch a standard remote desktop session.

If you do this you will be prompted to authenticate correctly.Firstly with your domain account to authenticate with the TMG RD gateway, then for other credentials to the test VM.

So a reasonable workaround, if a VPN or TMG Direct Access is not on option for you.