The blogs of Black Marble staff

In place upgrade times from TFS 2013 to 2015

There is no easy way to work out how long a TFS in place upgrade will take, there are just too many factors to make any calculation reasonable

  • Start and end TFS version
  • Quality/Speed of hardware
  • Volume of source code
  • Volume of work items
  • Volume of work item attachments
  • The list goes on….

The best option I have found to a graph various upgrades I have done and try to make an estimate based in the shape of the curve. I did this for 2010 > 2013 upgrades, and now I think I have enough data from upgrades of sizable TFS instances to do the same for 2013 to 2015.



Note: I extracted this data from the TFS logs using the script in this blog post it is also in my git repo 

So as a rule of thumb, the upgrade process will pause around step 100 (the exact number varies depending on your starting 2013.x release), time this pause, and expect the upgrade to complete in about 10x this period.

It is not 100% accurate, but close enough so you know how long to go for a coffee/meal/pub or bed for the night

Announcing release of my vNext build tasks as extensions in the VSTS/TFS Marketplace

In the past I have posted about the vNext TFS build tasks I have made available via my GitHub repo. Over the past few weeks I have been making an effort to repackage these as extensions in the new VSTS/TFS Marketplace, thus making them easier to consume in VSTS or using the new extensions support in TFS 2015.2

This it is an ongoing effort, but I pleased to announce the release of the first set of extension.

  • Generate Release Notes – generates a markdown release notes file based on work items associated with a build
  • Pester Test Runner – allows Pester based tests to be run in a build
  • StyleCop Runner – allows a StyleCop analysis to be made of files in a build
  • Typemock TMockRunner – used TMockrunner to wrapper MSTest, allowing Typemock test to be run on a private build agent

To try to avoid people going down the wrong path I intend to go back through my older blog posts on these tasks to update them to point at new resources.

Hope you find these tasks useful. If you find any log any issues on Github

Unblocking a stuck Lab Manager Environment (the hard way)

This is a post so I don’t forget how I fixed access to one of our environments yesterday, and hopefully it will be useful to some of you.

We have a good many pretty complex environments deployed to our lab hyper-V servers, controlled by Lab manager. Operations such as starting, stopping or repairing those environments can take a long, long time, but this time we had one that was quite definitely stuck. The lab view showed the many servers in the lab with green progress bars about halfway across but after many hours we saw no progress. The trouble is, at this point you can’t issue any other commands to the environment from within the Lab Manager console – it’s impossible to cancel the operation and regain access to the environment.

Normally in these situations, stepping from Lab Manager to the SCVMM console can help. Stopping and restarting the VMs through SCVMM can often give lab manager the kick it needs to wake up. However, this time that had no effect. We then tried restarting the TFS servers to see if they’d got stuck, but that didn’t help either.

At this point we had no choice but to roll up our sleeves and look in the TFS database. You’d be surprised (or perhaps not) at how often we need to do that…

First of all we looked in the LabEnvironment table. That showed us our environment, and the State column contained a value of Repairing.

Next up, we looked in the LabOperation table. Searching for rows where the DataspaceId column value matched that of our environment in the LabEnvironment table showed a RepairVirtualEnvironment operation.

In the tbl_JobSchedule table we found an entry where the JobId column matched the JobGuid column from the LabOperation table. The interval on that was set to 15, from which we inferred that the repair job was being retried every fifteen minutes by the system. We found another entry for the same JobId in the tbl_JobDefinition table.

Starting to join the dots up, we finally looked in the LabObject database. Searching for all the rows with the same DataspaceId as earlier returned all the lab hosts, environments and machines that were associated with the Team Project containing the lab. In this table, our environment row had a PendingOperationId which matched that of the row in the LabOperation table we found earlier.

We took the decision to attempt to revive our stuck environment by removing the stuck job. That would mean carefully working through all the tables we’d explored and deleting the rows, hopefully in the correct order. As the first part of that, we decided to change the value of the State column in the LabEnvironment table to Started, hoping to avoid crashing TFS should it try to parse all the information about the repair job we were about to slowly remove.

Imagine our surprise, then, when having made that one change, TFS itself cleaned up the database, removed all the table entries referring to the repair environment job and we were immediately able to issue commands to the environment again!

Net Writer: A great UWP blog editor

I came across Net Writer some months ago, when it's creator, Ed Anderson blogged about how he'd taken the newly-released Open Live Writer code and used it in his just-started Universal Windows Platform (UWP) app for Windows 10. In January it only supported blogger accounts, which meant that I was unable to use it. However, I checked again this weekend and discovered that it now supports a wide range of blog software including that powers

I'm writing this post using the app. It's great for quick posts (there's no plugin support so posting code snippets is tricky) and most importantly, it works on my phone! That's the big win as far as I'm concerned. I've been hankering for the ability to easily manage my blog form my phone for a long time and now I can.

You can find Net Writer in the Windows Store and learn more about it at Ed's blog.

Steps Required to Configure WSUS to Distribute Windows 10 Version 1511 Upgrade

Microsoft recently made a hotfix available that patches WSUS on Windows Server 2012 and 2012 R2 to allow Windows 10 upgrade to version 1511. Installing the update is not, however, the only step that is required…

  1. Install the hotfix. This can be downloaded from Ensure that you pick the appropriate hotfix for the version of Windows Server on which you’re running WSUS. Note that if you’re running Windows Server 2012 R2, there’s also a pre-requisite install.
  2. Once the hotfix is installed and you’ve restarted your WSUS server, look in the ‘Products and Classifications’ option under the Classifications tab and ensure that the checkbox for upgrades is selected. This is not selected automatically for you:
    Upgrades Option
    Note that the upgrade files may take quite some time to download to your WSUS server at the next synchronisation.
  3. Add a MIME-Type for ‘.esd application/octet-stream’ in IIS on the WSUS server. To do this:
    Open IIS Manager
    Select the server name
    From the ‘IIS’ area in the centre of IIS Manager, open ‘MIME Types’
    Click ‘Add…’
    Enter the information above:
    Esd MIME Type
    Click OK to close the dialog.
    Note: Without this step, clients will fail to download the upgrade with the following error:
    Installation Failure: Windows failed to install the following update with error 0x8024200D: Upgrade to Windows 10 [SKU], version 1511, 10586.
  4. Approve the Upgrade for the classes of computer in your organisation that you want to be upgraded.

Once all of the above steps are in place, computers that are targeted for the upgrade should have this happen automatically at the next update cycle.

New books on VSTS/TFS ALM DevOps

It has been a while since I have mentioned any had new books on TFS/VSTS, and just like buses a couple come along together.

These two, one from Tarun Arora and the other from Mathias Olausson and Jakob Ehn are both nicely on trend for the big area of interest for many of the companies I am working with at present; best practice ‘cook book’ style guidance on how to best use the tools in an ALM process.



If your are working with TFS/VSTS worth a look

Exchange 2013 Cert Change - Unable to Support the STARTTLS SMTP Verb

I saw a issue recently on an Exchange server after the certificate used to secure SMTP and IIS services was changed as the old certificate was about to expire.

The original default certificate that is self-generated had been replaced with one from a certificate authority. This had been used for several years without issue. The Exchange server is configured in hybrid mode, with incoming e-mail routed through Microsoft’s Exchange Online Protection (EOP) and TLS is configured to be required.

The actions taken were:

  1. Add the new certificate to the certificate store on the Exchange server. This made the new certificate available within the Exchange Admin Center on the server.
  2. Modify the services assigned to the new certificate to bind SMTP and IIS to the new certificate.
  3. Remove the original certificate from the server.
  4. Restart the Microsoft Exchange Transport Service on the server.

At this point, an error was thrown in the Application Event Log on the server and incoming mail from Exchange Online Protection stopped flowing. The error thrown was:

Log Name:      Application
Source:        MSExchangeFrontEndTransport
Date:          04/02/2016 12:17:20
Event ID:      12014
Task Category: TransportService
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <Exchange Server FQDN>
Microsoft Exchange could not find a certificate that contains the domain name <I><Cert Issuer Details><S><Cert Subject Details> in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector <Receive Connector Name> with a FQDN parameter of <I><Cert Issuer Details><S><Cert Subject Details>. If the connector's FQDN is not specified, the computer's FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.

I ran the suggested command to ensure that the Exchange Transport Service had access to the certificate key, but this didn’t help.

Restoring the soon-to-expire certificate to the certificate store on the server and restarting the Microsoft Exchange Transport Service fixed the error, however the certificate in question was going to expire soon, and the use of expired certificates for TLS to EOP is no longer allowed, so this didn’t really help much.

While digging into the configuration for the receive connector specified in the error thrown, I noticed something interesting. Despite the new certificate being supplied by the same certificate authority as the old one, the issuer specified for the certificate had changed slightly. The subject information was still the same. Sure enough, the properties of the receive connector in question still showed the old certificate details even through Exchange had been configured with the new certificate for SMTP and IIS. The information on the receive connector can be found by issuing the following command:

Get-ReceiveConnector "<Receive Connector Name>" | fl

The property we’re interested in is TlsCertificateName.

To correct the error, the following steps were taken:

  1. Locate the issuer and subject information from the new certificate. This can be done by examining the certificate directly via the certificate store, or using PowerShell, e.g.
    $certs = Get-ExchangeCertificate
    Locate the certificate you want to use. The one we wanted was the first on the list.
  2. Assemble the new issuer and subject information in a suitable format for the Receive Connector configuration. Again this can be done by copying the required text from the certificate information, or using PowerShell, e.g.:
    $certinfo = “<I>” + $certs[0].issuer + “<S>” + $certs[0].subject
  3. Modify the Receive Connector configuration to include the new certificate information assembled above, e.g.:
    Set-ReceiveConnector “<Receive Connector Name>” –TlsCertificateName $certinfo
  4. Restart the Microsoft Exchange Transport Service.
  5. Remove the old certificate from the server.

A quick test of incoming and outgoing mail indicated that everything was flowing as expected.

A vNext build task and PowerShell script to generate release notes as part of TFS vNext build.

Updated 22 Mar 2016: This task is now available as an extension in the VSTS marketplace

A common request I get from clients is how can I create a custom set of release notes for a build? The standard TFS build report often includes the information required (work items and changesets/commits associate with the build) but not in a format that is easy to redistribute. So I decided to create a set to tools to try to help.

The tools are available on my github account in two forms:

Both generate a markdown release notes file based on a template passed into the tool. The output report being something like the following:

Release notes for build SampleSolution.Master

Build Number: 20160229.3
Build started: 29/02/16 15:47:58
Source Branch: refs/heads/master

Associated work items
  • Task 60 [Assigned by: Bill <TYPHOONTFS\Bill>] Design WP8 client
Associated change sets/commits
  • ID bf9be94e61f71f87cb068353f58e860b982a2b4b Added a template
  • ID 8c3f8f9817606e48f37f8e6d25b5a212230d7a86 Start of the project

The Template

The use of a template allows the user to define the layout and fields shown in the release notes document. It is basically a markdown file with tags to denote the fields (the properties on the JSON response objects returned from the VSTS REST API) to be replaced when the tool generates the report file.

The only real change from standard markdown is the use of the @@TAG@@ blocks to denote areas that should be looped over i.e: the points where we get the details of all the work items and commits associated with the build.

#Release notes for build $defname  
**Build Number**  : $($build.buildnumber)   
**Build started** : $("{0:dd/MM/yy HH:mm:ss}" -f [datetime]$build.startTime)    
**Source Branch** : $($build.sourceBranch) 
###Associated work items 
* **$($widetail.fields.'System.WorkItemType') $($** [Assigned by: $($widetail.fields.'System.AssignedTo')] $($widetail.fields.'System.Title') 
###Associated change sets/commits 
* **ID $($csdetail.changesetid)$($csdetail.commitid)** $($csdetail.comment)   

Note 1: We can return the builds startTime and/or finishTime, remember if you are running the template within an automated build the build by definition has not finished so the finishTime property is empty to can’t be parsed. This does not stop the generation of the release notes, but an error is logged in the build logs.

Note 2: We have some special handling in the @@CSLOOP@@ section, we include both the changesetid and the commitid values, only one of there will contain a value, the other is blank. Thus allowing the template to work for both GIT and TFVC builds.

What is done behind the scenes is that each line of the template is evaluated as a line of PowerShell in turn, the in memory versions of the objects are used to provide the runtime values. The available objects to get data from at runtime are

  • $build – the build details returned by the REST call Get Build Details
  • $workItems – the list of work items associated with the build returned by the REST call Build Work Items
  • $widetail – the details of a given work item inside the loop returned by the REST call Get Work Item
  • $changesets – the list of changeset/commit associated with the build build returned by the REST call Build Changes
  • $csdetail – the details of a given changeset/commit inside the loop by the REST call to Changes or Commit depending on whether it is a GIT or TFVC based build

There is a file that just dumps out all the available fields in the PowerShell repo to help you find all the available options

Differences between the script and the task

The main difference between the PowerShell script and the build task is the way the connection is made to the REST API. Within the build task we pickup the access token from the build agent’s context. For the PowerShell script we need to pass credentials in some form or the other, either via parameters or using the default Windows credentials.



The script can be used in a number of ways

To generate a report for a specific build on VSTS

 .\Create-ReleaseNotes.ps1 -collectionUrl -teamproject "Scrum Project" –defname "BuildTest" -outputfile "" -templatefile "" -buildnumber "yourbuildnum" -password yourpersonalaccesstoken

Or for the last successful build just leave out the buildnumber

 .\Create-ReleaseNotes.ps1 -collectionUrl -teamproject "Scrum Project" –defname "BuildTest" -outputfile "" -templatefile "" -password yourpersonalaccesstoken

Authentication options

  1. VSTS with a personal access token – just provide the token using the password parameter
  2. If you are using VSTS and want to use alternate credentials just pass a username and password
  3. If your are using the script with an on-premises TFS just leave off both the username and password and the Windows default credentials will be used.

In all cases the debug output is something like the following

VERBOSE: Getting details of build [BuildTest] from server [ Project]
VERBOSE: Getting build number [20160228.2]
VERBOSE:    Get details of workitem 504
VERBOSE:    Get details of changeset/commit ba7e613388c06b8440c9e7601a8d6fa29d588051
VERBOSE:    Get details of changeset/commit 52570b2abb80b61a4a629dfd31c0ce071c487709
VERBOSE: Writing output file  for build [BuildTest] [20160228.2].

You should expect to get a report like the example shown at the start of this post.

Build Task

The build task needs to be built and uploaded as per the standard process detailed on my vNext Build’s Wiki (am considering creating a build extensions package to make this easier, keep an eye on this blog)

Once the tool is upload to your TFS or VSTS server it can be added to a build process


The task takes two parameters

  • The output file name which defaults to $(Build.ArtifactStagingDirectory)\
  • The template file name, which should point to a file in source control.

There is no need to pass credentials, this is done automatically

When run you should expect to see a build logs as below and a releases notes file in your drops location.



So I hope some people find these tools useful in generating release notes, let me know if they help and how they could be improved.

My Resource Templates from demos are now on GitHub

I’ve had a number of people ask me if I can share the templates I use in my Resource Template sessions at conferences. It’s taken me a while to find the time, but I have created a repo on GitHub and there is a new Visual Studio solution and deployment project with my code.

One very nice feature that this has enabled me to provide is the same ‘Deploy to Azure’ button as you’ll find in the Azure Quickstart Templates. This meant a few changes to the templates – it turns out that Github is case sensitive for file requests, for example, whilst Azure Storage isn’t. The end result is that you can try out my templates in your own subscription directly from Github!