But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

High CPU utilisation on the data tier after a TFS 2010 to 2013 upgrade

There have been significant changes in the DB schema between TFS 2010 and 2013. This means that as part of an in-place upgrade process a good deal of data needs to be moved around. Some of this is done as part of the actual upgrade process, but to get you up and running quicker, some is done post upgrade using SQL SPROCs. Depending how much data there is to move this can take a while, maybe many hours. This is the cause the SQL load.

A key factor as to how long this takes is the size of your pre upgrade tbl_attachmentContent table, this is where amongst other things test attachments are stored. So if you have a lot of test attachments it will take a while as these are moved to their new home in tbl_content.

If you want to minimise the time this takes it can be a good idea to remove any unwanted test attachments prior to doing the upgrade. This is done with the test attachment cleaner from the appropriate version of TFS Power Tools for your TFS server. However beware that if you don’t have a suitably patched SQL server there can be issues with ghost files (see Terje’s post).

If you cannot patch your SQL to a suitable version to avoid this problem then it is best to clean our old test attachments only after the while TFS migrate has completed i.e. wait until the high SQL CPU utilisation caused by the SPROC based migration has completed. You don’t want to be trying to clean out old test attachments at the same time TFS is trying to migrate them.

Speaking at Gravitas’s Tech Talk #3 -

I am speaking at Gravitas’s Tech Talk #3 - "TFS & Visual Studio 2013 vs The Rest" on Tuesday march the 4th about

"Microsoft's Application Lifecycle product has had a lot of changes in the past couple of years. In this session will look at how it can be used provide a complete solution from project inception through development, testing and deployment for project both using Microsoft and other vendor technologies"

Hope to see you there

Upgraded older Build and Test Controllers to TFS 2013

All has been going well since our upgrade from TFS 2012 to 2013, no nasty surprises.

As I had a bit of time I thought it a good idea to start the updates of our build and lab/test systems. We had only upgraded our TFS 2012.3 server to 2013. We had not touched our build system (one 2012 controller and 7 agents on various VMs) and our Lab Management/test controller. Our plan, after a bit of thought, was to so a slow migration putting in new 2013 generation build and test controllers in addition to our 2012 ones. We would then decide on an individual build agent VM basis what to do, probably upgrading the build agents and connecting them to the new controller. There seems to be no good reason to rebuild the whole build agent VMs with the specific SDKs and tools they each need.

So we created a new pair of Windows 2012R2 domain joined server VMs, on  one we installed a test controller and on the build controller and a single build agent

Note: I always tend to favour a single build agent per VM, usually using a single core VM. I tend to find most builds are IO locked not CPU locked so having more smaller VMs, I think, tends to be easier to manage at the VM hosting resource level.

Test Controller

Most of the use of our Test Controller is as part of our TFS Lab Management environments. If you load MTM 2013 you will see that it cannot manage 2012 Test Controller, they appear as off line. Lab Management is meant to keep the test agents upgraded, so it should upgrade an agent from one point release to another e.g. 2012.3 to 2012.4. However, this upgrade feature does not extend to 2012 to 2013 major release upgrades. Also I have always found the automated deployment/upgrade of test agents as part of an environment deployment is problematic at best. You often seem to suffer DNS and timeout issues. Easily the most reliable method is to make sure the correct (or at least compatible) test agents are installed on all the environment VMs prior to their configuration at the deployment/restart stage.

Given this the system that seems to work for getting environment’s test agents talking to the new 2013 Test Controller is:

  1. In MTM stop the environment
  2. Open the environment settings and change the controller to the new one
  3. Restart the environment, you will see the VMs show as not ready. The test agents won’t configure.
  4. Connect to each VM
    1. Uninstall the 2012 Test Agent
    2. install the 2013 Test Agent
  5. Stop and restart the environment and all should work – with luck the VMs will configure properly and show as ready
  6. If the don’t
    1. Try a second restart, sometimes sorts it
    2. You can try a repair, re-entering the various password.
    3. Updated 5 Feb 2014 Found I always need to do this. If problem really persist try running  Run the Test Agent Configuration tool on each VM, press next,next, next etc. and it will try to configure. It will probably fail, but hopefully it will have done enough port opening etc. to allow the next environment restart to work correctly
    4. If it still fails you need to check the logs, but suspect a DNS issue.

Obvious you could move step 4 to the start if you make the fair assumption it is going to need manual intervention

Build Controller

Swapping your build over from 2012 to 2013 will have site specific issues. It all depends what build activities you are using. If they are bound to TFS 2012 API they may not work unless you rebuild them. However from my first tests I have found my Build 2012 processes template seem to work I, this is whether I set my  build controller ‘custom assemblies path’ to either my 2012 DLL versions or their 2013 equivalents. So .NET is managing to resolve usable DLLs to get the build working.

Obviously there is still more to do here, checking all my custom build assemblies, maybe a look at revising the whole build scripts to make use to 2013 features, but that can wait.

What I have now allows me to upgrade our Windows 8.1 build agent VM so it can connect to our 2013 Build Controller. Thus allowing use to run full automated builds and tests of Windows 8.1 application. Up to now with TFS 2012 we had only been able to the basic build working due to having to hack i the build process as you need Visual Studio 2013 generation tools to fully build and test Windows 8.1 applications.

 

So we are going to have 2012 build and test controllers around for  while, but we have provided the migration is not going to be too bad. Maybe just needs a bit of thought over some custom build assemblies.

How long is my TFS 2010 to 2013 upgrade going to take?

Update 27 Jun 2013 See update version of post with more data

I seem to be involved with a number of TFS 2010 to 2013 upgrades at present. I suppose people are looking at TFS 2013 in the same way as they have historically looked at the first service pack for a product i.e: the time to upgrade when most of the main issues are addressed. That said TFS 2013 is not TFS 2012 SP1!

A common question is how long will the process take to upgrade each Team Project Collection? The answer is that it depends, a good consultants answer. Factors include the number of work items, size of the code base, number of changesets, volume of test results and the list goes on.  The best I have been able to come up with is to record some timings of previous upgrades and use this data to make an educated guess. 

In an upgrade of a TPC from TFS 2010 to 2013 there are 793 steps to be taken. Not all these take the same length of time, some are very slow as can be seen in the chart. I have plotted the points where the upgrade seems to pause the longest. These are mostly towards the start of the process where I assume the main  DB schema changes are being made

image

To give some more context

  • Client C was a production quality multi tier setup and took about 3 hours to complete.
  • Client L, though with a a similar sized DB to Server A, was much slower to upgrade, around 9 hours. However, it was on a slower single tier test VM and also had a lot of historic test data attachments (70%+ of the DB contents)
  • Demo VM was my demo/test TFS 2010 VM, this had 4 TPCs, the timing are for the largest of 600Mb. In reality this server had little ‘real’ data. It is also interesting to note that though there were four TPCs the upgrade did three in parallel and when the first finished started the fourth. Worth remembering if you are planning an upgrade of many TPCs.

Given this chart, if you know how long it takes to get to Step 30 of 793 you can get an idea of which of these lines closest matches your system.

I will continue to update this post as I get more sample data, hope it will be of use to others to gauge how only upgrades may take, but remember your mileage may vary.

Upgrading our TFS 2012 server to 2013

We have eventually got around to the 2013 upgrade of our production TFS server. It had been put off due to some tight delivery deadlines around Christmas.

The upgrade went fine, unlike out some previous ones we have had.

The upgrading of our team process templates, to add the new features, was greatly eased by using Feature4Tfs tool on CodePlex. This meant one command line call and all the projects were done (we had no significant process customisation) as opposed to visiting in team project in the admin console.

For now we are continuing to run with our TFS 2012 generation build and test controllers. These are working fine with 2013, so we can upgrade these when it is convenient, not all in a rush.

Fix for intermittent connection problem in lab management – restart the test controller

Just had a problem with a TFS 2012 Lab Management deployment build. It was working this morning, deploying two web sites via MSDeploy and a DB via a DacPac, then running some CodedUI tests. However, when I tried a new deployment this afternoon it kept failing with the error:

The deployment task was aborted because there was a connection failure between the test controller and the test agent.

image

If you watched the build deployment via MTM you could see it start OK, then the agent went off line after a few seconds.

Turns out the solution was the old favourite, to reboot of the Build Controller. Would like to know why it was giving this intermittent problem though.

Update 14th Jan An alternative solution to rebooting is to add a hosts file entry on the VM running the test agent for the IP address of the test controller. Seems the problem is name resolution, but not sure why it occurs