But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

More in rights being stripped for the [team project]\contributors group in TFS 2012 when QU1 applied and how to sort it.

I recently wrote a post that discussed how the contributor rights had been stripped off areas in TFS 2012 server when QU1 was applied, this included details on the patches to apply the manual steps to resolve the problem.

Well today I found that it is not just in the area security you can see this problem. We found it too in the main source code repository. Again the [Team project]\contributors group was completely missing. I had to re-add it manually. Once this was done all was OK for the users

image

FYI: You might ask how I missed this before, most of the  users  on this project had higher levels of rights granted by being members of other groups. It was not until someone was re-assigned between team we noticed.

Visual Studio 2012.2 changes

The Git support was not the only announcement for TFS at the ALM Summit last week. On Brian Harry’s blog you can see more on the new features either in the TFS/VS 2012.2 (Update 2) CTP or planned to appear in later CTPs. The list is long, but the ones that caught my eye beyond that of Git support are

  • Microsoft Fakes moves from the Ultimate SKU to Premium, thus making it a ‘free’ option for many corporate developers as they already have that SKU
  • Easier customisation of the Kanban board
  • Tagging of workitems to allow flexible filtering
  • Testing visibility in the web admin pages

… and many other improvements. Have a look at the blog posts, or even better pull down the CTP for a look. Also remember if you want to see new TFS features why not try them via a Team Foundation Server account?

My TFS session at Black Marble’s Tech Update is already out of date, there were announcements last night

At the ALM Summit yesterday Brian Harry made some major TFS and Visual Studio announcements

  • Git support for the hosted visualstudio.com, this allows you to choose if you want a centralised (existing TFS) source control repository or DVSC (using Git). There is also new tools with VS2012 to make using Git easier. Read more as to why Microsoft have made this addition to their offering in his blog. For those of you using on premises TFS you will have have to wait for the next major release of TFS, don’t expect to see this in a quarterly update.
  • Also he outline  what is to be in Visual Studio 2012 Update 2, loads of tool enhancements.

Have a look the posts to find out more

Fixing area permission issues when creating new teams in TFS 2012 after QU1 has been installed

[Updated 4 Fe 2013 See http://blogs.msdn.com/b/bharry/archive/2013/02/01/hotfixes-for-tfs-2012-update-1-tfs-2012-1.aspx for the latest on this ]

One of the side effects of the problems we had with TFS 2012 QU1 was that when we created a new team within a team project contributors had no rights to the teams default Area. The workaround was that we had to add these rights manually, remembering to add these as you would expect is something you forget all the time, so it would be nice to fix the default.

The solution it turns out is straight forward, any new team gets the area rights inherited from the default team/root of the team project.

  1. Open the TFS web based control panel 
  2. Select the Team Project Collection
  3. Select the Team Project
  4. Select the ‘Areas’
  5. Select the root node (has the same name as the Team Project)
  6. Using the drop down menu to the left of the checkbox, select security
  7. Add the Contributor TFS Group and grant it the following rights

image

These settings will be used as the template for any new teams created with the Team Project.

TF237111 errors when trying to add work items to the backlog after TFS 2012 QU1 is applied

[Updated 4 Feb 2013 See http://blogs.msdn.com/b/bharry/archive/2013/02/01/hotfixes-for-tfs-2012-update-1-tfs-2012-1.aspx for the latest on this ]

I posted earlier in the week about my experiences with the post TFS 2012 QU1 hotfix. When I posted I thought we had all our problems sorted, we did for new team projects, but it seems still had an issue for teams on our team projects that were created prior to the upgraded from RTM to QU1. As I said in the past post we got into this position due to trying to upgraded a TPC form RTM to QU1 by detaching from the 2012 RTM server and attaching to a 2012 QU1 server – this is not the recommended route and caused us to suffer the problem the KB2795609 patch addresses.

The problem we still had was follows:

  • I have two users a Team  Project called ‘BM’ who are in the team called ‘Bad TP’
    • Richard (the Team project creator and administrator)
    • Fred (a Team Project contributor)
  • All is fine for Richard, he can see the team’s product backlog and add items to it.
  • Fred can get to the team backlog page in the web client, but cannot see any work items and gets a TF237111 error if they try to add a new work item

image

  • The quick fix was to make Fred a team project administrator, but not a long term solution
  • We checked the following rights
    • Richard was a member of basically all the groups on the ‘BM’ team project (he was the creator so that was expected), the important ones were [BM\Project administrators, [BM]\contributors and ‘Bad TP’
    • Fred was a member of the [BM]\contributors  and ‘Bad TP’ team

clip_image001[6]

    • The ‘Bad TP’ team had the following permissions

clip_image001

So all these permissions looked OK as you would expect. What I had forgotten was that the team model in TFS 2012 is build around the Area’s hierarchy. This has security permissions too. To check this

  • Go to the Admin page for ‘Bad TP’
  • Click the “Areas” tab
  • Right click the “default area” for the team and select “security”
  • We had expect to see some like this

image

  • However there was no entry at all for the Contributors group.
  • I added this in and had to explicitly set the four ‘inherited allow‘ permissions to ‘allow’ and everything started to work.

So the problem was that during the problematic upgraded we had managed to strip off all the contributor group entries from area in the existing Team Project. The clue was actually in the TF237111 error as this does mention permissions are the area path.

So now we know we can fix the issue. It should be noted that any new teams created in the team project seem to not get this right applied, so we have to remember to added it when we create a new team.

Incorrect IIS IP Bindings and TFS Server Url

By default the TFS server uses http://localhost:8080/tfs as it’s Server URL, this is the URL used for internal communication, whereas the Notification Url is the one TFS tells client to communicate to it via. Both these Urls can be changed via the Team Foundation Server Console, but I find you do not usually need to change the Server Url, only the notification one.

image

I hit a problem recently on a site where if you tried to edit the Team Project Collection Group Membership (via the web or TFS admin console) you got a dialog popping up saying  ‘HTTP 400 error’. Now this you have to say looks like a URL/binding issue, the tools cannot find an end point.

Turns out the issue was that there had been a IP addressing schema changes on the network. The different services on the network had been assigned their own IP addresses (as well as the host having its own IP address) e.g. On our TFS server we might have

  • 10.0.0.1 – physicalservername.domain.com
  • 10.0.1.1 – tfs2012.domain.com
  • 10.0.1.2 – sharepoint.domain.com

This is all well end good, but a mistake had been made in the bindings in IIS during the reconfiguration.

image

The HTTPS bind was correct the hostname matched the IP address, this has to be the case else SSL does not work. However, the HTTP port 8080 should have been bound  to all IP Addresses (i.e. no hostname and the * IP address as above). On the site, HTTP was bound to a specific IP address. This was fine if a client connected to http://tfs2012.domain.com:8080/tfs (which resolved to the correct address), but failed for http://loclahost:8080/tfs  as the binding did not match.

Once the edit was made to remove the hostname all was OK (the other option would have been to alter the server Url to match)

So problem fixed, the strangest thing is that this issue only appeared to effect setting TPC group membership, everything else was fine.

Experiences applying TFS 2012 QU1 and it subsequent hotfix

Brian Harry posted last week about a hotfix for TFS 2012 QU1 (KB2795609). This should not be needed by most people, but as his post points out does fix issues for a few customers. Well we were one of those customers. When upgrading from 2012 RTM to 2012 QU1 we had attempted what with hindsight was an over ambitious hardware migration too. This involved swapping our data tier from a SQL 2012 instance to a new 2012 availability group and merging team project collections from different server as well as applying the QU1. Our migration plan contained some team project collection detach/attach steps hence getting into the area this hotfix addresses.

The end point was we ended up with a QU1 upgraded server, but we could only get users connected if we made them team project administrators, a valid short term solution, but something we needed to fix.

We therefore applied the new KB2795609 patch, but hit a gotcha that you should be aware of

  • We ran the patch EXE on our TFS server that was showing the problem.
  • This ran without error, taking about 5 minutes
  • We tried to connect to the patched TFS server via the web client and VS2012, we could make a connection to TFS but could open any TPCs
  • On checking the TFS admin console we saw the TPC was offline and reporting that the servicing had failed (but this had not been reported back via the patch tool)
  • We reran the servicing job (via the TFS admin console) but it failed in the core step we saw in the logs

[Error] TF400744: An error occurred while executing the following script: TurnOnRCSI.sql. Failed batch starts on the line 1. Statement line: 1. Script line: 1. Error: 5069 ALTER DATABASE statement failed.

  • Our TFS DBs are now stored with a SQL 2012 availability group, during the upgrade to QU1 we had seen problems applying the upgrade unless we removed the DBs from the availability groups. So we removed the tfs_configuration and tfs_[mytpc] from availability groups and re applied the servicing job and all was OK
  • Once the servicing of the TPC was completed it went online as expected.
  • We then put the DBs back into the availability group
  • We could then remove the users from the team project administrators group as their previous rights were working again.

So we now had a patched and working TFS 2012 QU1 server. Lets hope that QU2 is a little smoother and we don’t need the direct help of product group, who I must say have been great in getting this problem addressed. I really like the openness we see in Brian’s blog of both the good and the bad.

Why can’t I create an environment using a running VM on my Lab Management system?

With TFS lab management you can build environments from stored VM and VM templates stored in an SCVMM library or from VMs running on a Hyper-V host within your lab infrastructure. This second form is what used to be called composing an environment in TFS 2010. Recently when I tried to compose an environment I had a problem. After selecting the running VM inside the new environment wizard I got the red star that shows an error in the machine properties

image

Now I would only expect to see this when creating an environment with a VM templates as a red star usually means the OS profile is not set e.g. you have missed a product key, or passwords don’t match. However, this was a running VM so there were no settings I could make, and no obvious way to diagnose the problem. After a few email with Microsoft Lab management team we go to the bottom of the problem, it was all down to the Hyper-V hosts network connections, but that is rushing ahead, first lets see why it was a confusing problem.

First the red herring

We now know the issue was the Hyper-V host network, but at first it looked like I could compose some guest VMs but not others. I wrongly assumed the issue was some bad meta-data or corrupt settings within the VMs. Tthis problem all started after a server crash and so we were fearing corruption, which clouded our thoughts.

The actual reason some VMs could be composed and some could not was dependant on which Hyper-V host they were running on. Not the VMs themselves.

The diagnostic steps

To get to the root of this issue a few commands and tools were used. Don’t think for a second there was not a lot of random jumping about and trial and error. In this post I am just going to point out what was helpful.

Firstly you need to use the TFSConfig command on your TFS server to find out your network location setting. So run

C:\Program Files\Microsoft Team Foundation Server 11.0\Tools>tfsconfig lab /settings /list
SCVMM Server Name: vmm.blackmarble.co.uk
Network Location: VSLM Network Location
IP Block: 192.168.23.0/24
DNS Suffix: blackmarble.co.uk

Next you need to see which, if any, of your Hyper-V hosts are connected to this location. You can do this in a few graphically ways in SCVMM (and I am sure via PowerShell too)

If you select a Hyper-V host in SCVVM, right click and select View networking. On a healthy host you see the VSLM network location connected to external network adaptor the VMs are using

image

On my failing Hyper-V host the VSLM network was connected to an empty network port

image

You can also see this on the SCVMM > host (right click) > properties. If you look on  the networking tab for the main virtual network  you should see the VSLM network as the location. On the failing Hyper-V host this location was empty.

image

The solution

You would naively think selecting the edit option on the screen shot above would allow you to enter the VSLM Network as the location, but no. Not on that tab. You need to select the hardware tab.

image

You can then select the correct network adaptor and override the discovered network location to point to the VSLM Network Location. Once this was done I could compose environments as I would expect.

I have said it before, but Lab Management has a lot of moving parts, and they all must be setup right else nothing works. A small configuration error can seriously ruin your day.

Did I delete the right lab?

It was bound to happen in the end, the wrong environment got deleted on our TFS Lab Management instance. The usual selection of rushing, minor mistakes, misunderstandings and not reading the final dialog properly and BANG you get that sinking feeling as you see the wrong set of VMs being deleted. Well this happened yesterday, so was there anything that can be done? Luckily the answer is yes, if you are quick.

Firstly we knew SCVMM operations are slow, so I RDP’d onto the Hyper-V host  and quickly copied the folders that contained the VMs scheduled to be deleted. We now had a copy of the VHDs.

On the SCVMM host I cancelled the delete jobs. Turns out this did not really help as the jobs just get rescheduled. In fact it may make matters worse as the failing of jobs and their restarting seems to confuse SCVMM, took it hours before it was happy again, kept giving ‘can’t run job as XXX in use’ and losing sight of the Hyper-V hosts (needed to restart the VMM service in the end).

So I now had a copy of three network isolated VM, so I

  • Created new VMs on a Hyper-V host using Hyper-V manager with the saved VHDs as their disks. I then made sure they ran and were not corrupted
  • In SCVMM cleared down the saved state so they were stopped (I forgot to do this the first time I went through this process and it meant I could not deploy the stored VMs into an isolated environment, that wasted hours!)
  • In SCVMM put them into the library on a path our Lab Management server knows about (gotcha here is SCVMM deletes the VM after putting it into the library, this is unlike MTM Lab Center which leaves the original in place, always scares me when I forget)
  • In MTM Lab Center import the new VMs from the library
  • Create a new network isolated environment with the VMs
  • Wait……………………….

When it eventually started I had a network isolated environment back to the state it was when we in effect pulled the power out. All took about 24 hours, but most of this was waiting for copies to and from the library to complete.

So the top tip is try to avoid the problem, this is down to process frankly

  • Use the ‘mark a in use’ feature to say who is using a VM
  • Put a process in place to manage the lab resources. It does not matter how much Hyper-V resource you have you will run out in the end and be unable to add that extra VM. You need a way to delete/archive out what is not currently need
  • Read the confirmation dialogs, they are there for a reason