But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

401.1 Permission error with on-premises TFS when accessing the API with a PAT

Background

If you are creating VSTS build extensions you will need to get the build or release’s PAT token if you wish to call the VSTS REST API.

This is done using a call like this (Node)

 

import tl = require('vsts-task-lib/task');

var auth = tl.getEndpointAuthorization('SYSTEMVSSCONNECTION', false);

if (auth.scheme === 'OAuth') {

var token = auth.parameters['AccessToken'];

 

or (PowerShell)

 

$vssEndPoint = Get-ServiceEndPoint -Name "SystemVssConnection" -Context $distributedTaskContext

$personalAccessToken = $vssEndpoint.Authorization.Parameters.AccessToken

 

You pop the resultant PAT into the headers of your REST web request and you are away and running.

The Problem

I hit a problem using this logic on VSTS Extension when they are run on TFS. On VSTS all was fine, but on TFS I got an unexpected 401.1 permission error on the first REST call i.e. I could not access the VSTS REST API

I tried setting fiddling with rights of my build user account, it was not that. Also I tried setting the ‘Allow scripts to access OAuth token’ property for the build/release agent

 

image

But this does not help either. This option just makes the PAT available as an environment variable, so you don’t need to use the code shown above.

And anyway – it was all worked on VSTS so it could not have been that!

Solution

The answer was I had basic authentication enabled on my Test TFS VM, as soon as this is disabled (the default) everything leapt into life.

image

Presenting at an event in Leeds - Making it easy to migrate your ALM process to the Cloud

Do you find your TFS server gets forgotten?

  • It is not owned by the IT department and the Development team don’t have the time to support it fully, it never gets patched or upgrades?
  • Or maybe you are adopting a cloud first strategy for all you systems?

Well maybe it is time to consider moving your on-premises TFS instance to VSTS?

On the 9th of May at the Crowne Plaza Hotel in Leeds I will be presenting at a Black Marble /Microsoft event where we will be looking at Microsoft’s new high fidelity VSTS database import tools that can be used to move a TFS instance to VSTS.

I will also be considering the pros and cons of the other migration options available to you. Hopefully making this a very useful session if you are considering a move to VSTS from TFS or any other source control ALM solution.

Hope to see you there, to register click here

 

image

New Cross Platform version of my Generate Release Notes VSTS Extension

My Generate Release Notes VSTS extension has been my most popular by a long way. I have enhanced it, with the help of others via pull requests, but there have been two repeating common questions that have not been resolved

  1. Is it cross platform?
  2. Why does it show different work items and commit associations to the VSTS Release Status UI?

Well the answer to the first is that the core of the logic for the extension came from a PowerShell script we used internally, so PowerShell was the obvious first platform, especially as though my PowerShell skills are not great, my Node was weaker!

The second issue is due to my original extension and VSTS’s UI doing very different things. My old extension was based around inspecting build results, so when working in a release it finds all the builds between the current release and last successful one and looks at the details of each build in turn, building a big list or changes. VSTS’s Release summary UI does not do this, it make a few current undocumented ‘compare this to that’ API calls to get the lists.

In an attempt to address both these questions I have over the past few weeks created a new Cross Platform Generate Release Notes Extension. Now don’t worry, the old one is still there and supported, they do different jobs. This new extension is cross platform and tries to use the same API calls the VSTS Release summary UI uses.

There are of course a few gotchas

  • I did have to adopt a work around for TFVC changeset history as Microsoft use an old internal API call, but that that was the only place I had to do this. So apologies if there are any differences in the changesets returned.
  • The template format is very similar to that used in my original Generate Release Notes VSTS extension, but due to the change from PowerShell to Node I had to move from $($widetail.fields.'System.Title') style to ${widetail.fields['System.Title']}

So I hope people find this new extension useful, I can now go off happily closing old Issues in GitHub

VSTS Build Task Development - Now that is a misleading error message !

I have been working on a new Node.JS based VSTS Build Extension, so loads of Node learning going on as it is a language I have not done much with in the past. I expect to get caught out in places, but I have just wasted a good few hours on this one!

I am working in VS Code using Typescript to generate my Node based task. Whilst hacking around I have been just running the script in the VS Code debugger, getting the logic right before packaging it up and testing within a VSTS Build extension.

Everything was working until I did a big refactor, basically moving functions to a module. I suddenly started getting the following error trying to make a REST call

Exception occured

Error: Error: Hostname/IP doesn't match certificate's altnames: "Host: richardfennell.vsrm.visualstudio.com. is not in the cert's altnames: DNS:*.vsrm.visualstudio.com, DNS:vsrm.visualstudio.com, DNS:*.app.vsrm.visualstudio.com, DNS:app.vsrm.visualstudio.com"

image

 

I looked for ages to see how I had broken the REST call, all to no avail. In the end I rolled back and had to step through the refactor in small steps (smaller steps I should probably have taken anyway)

In the end I found the issue. The problem was in my early testing I had hard coded my input parameters e.g.

var templateFile = "template.md”;

Whilst stating to wire up the code as a VSTS Task I had started to swap in calls to the VSTS task Library

import tl = require('vsts-task-lib/task');

Correction – all the tl.xxx calls seem to cause a problem, avoid them for local testing

Now for items such as logging this works fine whether the logic is running in VS Code’s debugger or on a VSTS Build Agent, so I could use the following line in VS Code or on a VSTS Build Agent.

tl.Debug(“My Message”);

Where it does not work is for Task inputs. I had assume that

var templateFile = tl.getInput("templatefile");

Would return null/undefined when running in the VS Code debugger, but no, it causes that strange exception.

Once I removed the all the getInput calls my error went away.

Hope this save someone else some time

Migrating projects from CodePlex to GitHub due to CodePlex shutting down at the end of year

It has just been announced by Microsoft that it’s Open Source service CodePlex is shutting down before the end of the year. The reality is that the Microsoft focused Open Source community, and Microsoft itself, have moved to GitHub a good while ago.

I think I, like most developers, have moved any active Open Source projects to GitHub a good while ago,  but I still had legacy ones on CodePlex.

Microsoft have provided a nicely documented process to move the key assets of projects, whether TFVC or Git based, to GitHub. This process worked for me. However, I will suggest a could of changes/additions

  1. I would not export the WIKI docs as detailed in the process. I don’t want my old CodePlex Wiki pages in the new GitHub code repository as a folder. I think it is better to move each page over to a GitHub WIki. I only had few pages, so I did this by hand. I used this nice little tools  from Dom Christie to convert the CodePlex HTML based pages to Markdown which I cut and pasted into the new repo’s Wiki, fixing URLs as I went.
  2. I decided I needed to consider release downloads. The process does not do address this area. I thought I should bring over at least the last set of release binaries for my projects as a Github Releases. The reason was that the chances are for any old inactive project on CodePlex you won’t have the tools to hand to re-build the code easily, so just in case it is best to keep the last built version to hand as a release
  3. The process does not bring over Issues, but this was not a problem for me, the projects I have been superseded by active ones already on Github, so the issues are irrelevant

So if you have old CodePlex projects and you don’t want them to disappear think about moving them before the service is shutdown, you have until December 2017.

You never know how people will use a tool

You never know how people will use a tool once it is out ‘in the wild’. I wrote my Generate Release Notes VSTS extension to generate markdown files, but people have attempted to use it in other ways.

I realised, via an issue raised on Github, that it can also be used, without any code changes, to generate other formats such as HTML. The only change required is to provide an HTML based template as opposed to markdown one.

I have added suitable samples to the wiki and repo

Can’t add users to a VSTS instance backed by an Azure Directory

I have a VSTS instance that is backed by an Azure Directory. This is a great way to help secure a VSTS instance, only users in the Azure Directory can be added to VSTS, not just any old MSA (LiveIDs). This is a directory that can be shared with any other Azure based services such as O365, and centrally managed and linked to an on-premises Active Directory.

When I tried to add a user to VSTS, one that was a valid user in the Azure Directory, their account did not appear in the available users drop down.

 image

Turns out the problem was who I was logged in as. As yo can see from the screenshot I have three Richard accounts in the VSTS instance (and Azure Directory), a couple of MSAs and a guest work account from another Azure Directory. I was logged in as the guest work account.

All three IDs as administrators in VSTS, but it turned out I needed to be logged in as the MSA that owned the Azure subscription contains the Azure Directory. As soon as I used this account the dropdown populated as expected and I could add the users from the Azure Diretcory

image

Version 2.0.x of my Generate Release Notes VSTS Task has been released with release rollup support

I have just released a major update to my Generate Release Notes VSTS Build extension. This V2 update adds support to look back into past releases to find when there was a successful release to a given stage/environment and creates a rollup set of build artifacts, and hence commits/changesets and workitems, in the release notes.

 

 

This has been a long running request on GitHub for this extension which I am pleased to have been able to address.

To aid backwards compatibility, the default behaviour of the build/release tasks is as it was before, it can be used in a build or in and release, and if in a release it only consider the artifacts in the current release that ran the task.

If you want to use the new features you need to enable them. This is all on the advanced properties

 

image

 

You get new properties to enable scanning past releases until the task find a successful deployment to, by default, the same stage/environment that is currently being released too. You can override this stage name to allow more complex usage e.g. generating the releases notes for what is changed since the last release to production whist in a UAT environment.

This change also means there is new variable that can be accessed in templates, this $Releases which contains all the releases being used to get build artifacts. This can be used on release notes to show the releases being used e.g.

 

**Release notes for release $defname**
**Release Number**  : $($release.name)   
**Release completed** $("{0:dd/MM/yy HH:mm:ss}" -f [datetime]$release.modifiedOn) **Changes since last successful release to '$stagename'**  
**Including releases:**  
$(($releases | select-object -ExpandProperty name) -join ", " )  

 

Generating a content

 

Release notes for release Validate-ReleaseNotesTask.Master
Release Number : Release-69 
Release completed 05/01/17 12:40:19
Changes since last successful release to 'Environment 2' 
Including releases: 
Release-69, Release-68, Release-67, Release-66 

 

Hope you find this extension useful

A nice relaxing Christmas break (and by the way I migrated our on-premises TFS to VSTS as well)

Over the Christmas break I migrated our on premises TFS 2015 instance to VSTS. The reason for the migration was multi-fold:

  • We were blocked on moving to TFS 2017 as we could not easily upgrade our SQL cluster to SQL 2014
  • We wanted to be on the latest, greatest and newest features of VSTS/TFS
  • We wanted to get away from having to perform on-premises updates every few months

To do the migration we used the public preview of the TFS to VSTS Migrator.

So what did we learn?

The actual import was fairly quick, around 3 hours for just short of 200Gb of TPC data. However, getting the data from our on-premises system up to Azure was much slower, constrained by the need to copy backups around our LAN and our Internet bandwidth to get the files to Azure storage, a grand total of more like 16 hours. But remember this was mostly spent watching various progress bars after running various commands; so I was free to enjoy the Christmas break, I was not a slave to a PC.

This all makes it sound easy, and to be honest the actual production migration was, but this was only due to doing the hard work prior to the Christmas break during the dry run phase. During the dry run we:

  • Addressed the TFS customisations that needed to be altered/removed
  • Sorted the AD > AAD sync mappings for user accounts
  • Worked out the backup/restore/copy process to get the TPC data to somewhere VSTS could import it from
  • Did the actual dry run migration
  • Tested the dry run instance after the migrate to get a list of what else needed addressing and anything our staff would have to do to access the new VSTS instance
  • Documented (and scripted where possible) all the steps
  • Made sure we had fall back processes in place if the migration failed.

And arguably most importantly, discovered how long each step would take so we could set expectations. This was the prime reason for picking the Christmas break as we knew we could have a number of days where there should be no TFS activity (we close for an extended period) hence de-risking the process to a great degree. We knew we could get the migration done over weekend, but a weeks break was easier, more relaxed, Christmas seemed a timely choice.

You might ask the question ‘what did not migrate?’

Well a better question might be ’what needed changing due to the migration?’

It was not so much items did not migrate, just they are handled a bit differently in VSTS. The list of areas we needed to address were

  • User Licensing – we needed to make sure your user’s MSDN subscription are mapped to their work IDs.
  • Build/Release Licensing – we needed to decide how many private build agents we really needed (not just spin up more on a whim as we had done with our on-premises TFS), they cost money on VSTS
  • Release pipeline – now these don’t migrate as of the time of writing, but I wrote a quick tool to get 95% of their content moved.  After using this tool we did then need to also edit the pipelines, re-entering ‘secrets’ which are not exported, before retesting them

But that was all the issues we had to address, everything else seems to be fine with users just changing the URL they connected to from on-premises to VSTS.

So if you think migrating your TFS to VSTS seems like a good idea, why not have a look at the blog post and video on  the Microsoft ALM Blog about the migration tool. Remember that this is a Microsoft Gold DevOps Partner led process, so please get in touch with us at Black Marble or me directly via this blog if you want a chat about the migrations or other DevOps service we offer.