But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Version 2.0.x of my Generate Release Notes VSTS Task has been released with release rollup support

I have just released a major update to my Generate Release Notes VSTS Build extension. This V2 update adds support to look back into past releases to find when there was a successful release to a given stage/environment and creates a rollup set of build artifacts, and hence commits/changesets and workitems, in the release notes.

 

 

This has been a long running request on GitHub for this extension which I am pleased to have been able to address.

To aid backwards compatibility, the default behaviour of the build/release tasks is as it was before, it can be used in a build or in and release, and if in a release it only consider the artifacts in the current release that ran the task.

If you want to use the new features you need to enable them. This is all on the advanced properties

 

image

 

You get new properties to enable scanning past releases until the task find a successful deployment to, by default, the same stage/environment that is currently being released too. You can override this stage name to allow more complex usage e.g. generating the releases notes for what is changed since the last release to production whist in a UAT environment.

This change also means there is new variable that can be accessed in templates, this $Releases which contains all the releases being used to get build artifacts. This can be used on release notes to show the releases being used e.g.

 

**Release notes for release $defname**
**Release Number**  : $($release.name)   
**Release completed** $("{0:dd/MM/yy HH:mm:ss}" -f [datetime]$release.modifiedOn) **Changes since last successful release to '$stagename'**  
**Including releases:**  
$(($releases | select-object -ExpandProperty name) -join ", " )  

 

Generating a content

 

Release notes for release Validate-ReleaseNotesTask.Master
Release Number : Release-69 
Release completed 05/01/17 12:40:19
Changes since last successful release to 'Environment 2' 
Including releases: 
Release-69, Release-68, Release-67, Release-66 

 

Hope you find this extension useful

A nice relaxing Christmas break (and by the way I migrated our on-premises TFS to VSTS as well)

Over the Christmas break I migrated our on premises TFS 2015 instance to VSTS. The reason for the migration was multi-fold:

  • We were blocked on moving to TFS 2017 as we could not easily upgrade our SQL cluster to SQL 2014
  • We wanted to be on the latest, greatest and newest features of VSTS/TFS
  • We wanted to get away from having to perform on-premises updates every few months

To do the migration we used the public preview of the TFS to VSTS Migrator.

So what did we learn?

The actual import was fairly quick, around 3 hours for just short of 200Gb of TPC data. However, getting the data from our on-premises system up to Azure was much slower, constrained by the need to copy backups around our LAN and our Internet bandwidth to get the files to Azure storage, a grand total of more like 16 hours. But remember this was mostly spent watching various progress bars after running various commands; so I was free to enjoy the Christmas break, I was not a slave to a PC.

This all makes it sound easy, and to be honest the actual production migration was, but this was only due to doing the hard work prior to the Christmas break during the dry run phase. During the dry run we:

  • Addressed the TFS customisations that needed to be altered/removed
  • Sorted the AD > AAD sync mappings for user accounts
  • Worked out the backup/restore/copy process to get the TPC data to somewhere VSTS could import it from
  • Did the actual dry run migration
  • Tested the dry run instance after the migrate to get a list of what else needed addressing and anything our staff would have to do to access the new VSTS instance
  • Documented (and scripted where possible) all the steps
  • Made sure we had fall back processes in place if the migration failed.

And arguably most importantly, discovered how long each step would take so we could set expectations. This was the prime reason for picking the Christmas break as we knew we could have a number of days where there should be no TFS activity (we close for an extended period) hence de-risking the process to a great degree. We knew we could get the migration done over weekend, but a weeks break was easier, more relaxed, Christmas seemed a timely choice.

You might ask the question ‘what did not migrate?’

Well a better question might be ’what needed changing due to the migration?’

It was not so much items did not migrate, just they are handled a bit differently in VSTS. The list of areas we needed to address were

  • User Licensing – we needed to make sure your user’s MSDN subscription are mapped to their work IDs.
  • Build/Release Licensing – we needed to decide how many private build agents we really needed (not just spin up more on a whim as we had done with our on-premises TFS), they cost money on VSTS
  • Release pipeline – now these don’t migrate as of the time of writing, but I wrote a quick tool to get 95% of their content moved.  After using this tool we did then need to also edit the pipelines, re-entering ‘secrets’ which are not exported, before retesting them

But that was all the issues we had to address, everything else seems to be fine with users just changing the URL they connected to from on-premises to VSTS.

So if you think migrating your TFS to VSTS seems like a good idea, why not have a look at the blog post and video on  the Microsoft ALM Blog about the migration tool. Remember that this is a Microsoft Gold DevOps Partner led process, so please get in touch with us at Black Marble or me directly via this blog if you want a chat about the migrations or other DevOps service we offer.

My TFSAlertsDSL project has moved to GitHub and become VSTSServiceHookDsl

Introduction

A while ago I create the TFSAlertsDSL project to provide a means to script responses to TFS Alert SOAP messages using Python. The SOAP Alert technology has been overtaken by time with the move to Service Hooks.

So I have taken the time to move this project over to the newer technology, which is supported both on TFS 2015 (onwards) and VSTS. I also took the chance to move from CodePlex to GitHub and renamed the project to VSTSServiceHookDsl.

Note: If you need the older SOAP alert based model stick with the project on CodePlex, I don’t intend to update it, but all the source is there if you need it.

What I learnt in the migration

Supporting WCF and Service Hooks

I had intended to keep support for both SOAP Alerts and Service Hooks in the new project, but I quickly realised there was little point. You cannot even register SOAP based alerts via the UI anymore and it added a lot of complexity. So I decided to remove all the WCF SOAP handling.

C# or REST TFS API

The SOAP Alert version used the older TFS C# API, hence you had to distribute these DLLs with the web site. Whilst factoring I decided to swap all the TFS calls to using the new REST API. This provided a couple of advantages

  • I did not need to distribute the TFS DLLs
  • Many of the newer function of VSTS/TFS are only available via the REST API

    Exposing JObjects to Python

    I revised the way that TFS data is handed in the Python Scripts. In the past I hand crafted data transfer objects for consumption within the Python scripts. The problem with this way of working is that it cannot handle custom objects, customised work items are a particular issue. You don’t know their shape.

    I found the best solution was to just return the Newtonsoft JObjects that I got from the C# based REST calls. These are easily consumed in Python in the general form

    workitem["fields"]["System.State"] 


    Downside is that this change does mean that any scripts you had created for the old SOAP Alert version will need a bit of work when you transfer to the new Service Hook version.

    Create a release pipeline

    As per all good projects, I created a release pipeline for my internal test deployment. My process was as follows

    • A VSTS build that builds the code from Github this
      • Complies the code
      • Run all the unit test
      • Packages as an MSDeploy Package
    • Followed by a VSTS release that
      • Sets the web.config entries
      • Deploys the MSDeploy package to Azure
      • Then uses FTP to uploaded DSL DLL to Azure as it is not part of the package

    image 

    Future Steps

    Add support for more triggers

    At the moment the Service Hook project supports the same trigger events as the old SOAP project, with the addition of support Git Push triggers.

    I need to add in the handlers for all the older support triggers in VSTS/TFS, specifically the release related ones. I suspect these might be useful.

    Create an ARM template

    At the moment the deployment relies on the user creating the web site. It would be good to add an Azure Resource Management (ARM) Template to allow this site to be created automatically as part of the release process

    Summary

    So we have a nice new Python and Service Hook based framework to help manage your responses to Service Hook triggers for TFS and VSTS.

    If you think it might be useful to you why not have a look at https://github.com/rfennell/VSTSServiceHookDsl.

    Interested to hear your feedback  

  • Transform tool for transferring TFS 2015.3 Release Templates to VSTS

    If you are moving from on-premises TFS to VSTS you might hit the same problem I have just have. The structure of a VSTS releases is changing, there is now the concept of multiple ‘Deployment Steps’ in an environment. This means you can use a number of different agents for a single environment – a good thing.

    The downside this that if you export a TFS2015.3 release process and try to import it to VSTS it will fail saying the JSON format is incorrect.

    Of course you can get around this with some copy typing, but I am lazy, so….

    I have written a quick transform tool that converts the basic structure of the JSON to the new format. You can see the code as Github Gist

    It is a command line tool, usage is as follows

    1. In VSTS create a new empty release, and save it
    2. Use the drop down menu on the newly saved release in the release explorer and export the file. This is the template for the new format e.g. template.json
    3. On your old TFS system export the release process in the same way to get your source file e.g. source.json
    4. Run the command line tool providing the name of the template, source and output file

      RMTransform template.json source.json output.json
    5. On VSTS import the newly create JSON file release file.
    6. A release process should be created, but it won’t be possible to save it until you have fixed a few things that are not transferred
      1. Associated each Deployment step with Agent Pool
      2. Set the user accounts who will do the pre-and post approvals
      3. Any secret variable will need to be reentered
        IMPORTANT - Make sure you save the imported process as soon as you can (i.e. straight after fixing anything that is stopping it being saved). If you don't save and start clicking into artifacts or global variable it seems to loose everything and you need to re-import

    image

    It is not perfect, you might find other issues that need fixing, but it save a load of copy typing

    Deleting unwanted orphan XAML Build Controllers on a migrated VSTS instance

    Whilst working with the VSTS Data Import Service I ended up migrating a TFS TPC up to VSTS that had an old XAML Build Controller defined. I did not need this XAML build controller, in fact I needed to remove it because it was using my free private build controller slot. Problem was I could not find a way to remove it via the VSTS (or Visual Studio Team Explorer) UI, and the VM that had been running the build controller was long gone.

    The way I got rid of it in the end was the TFS C# API and a quick command line tool as shown below.

    Note that you will need to delete any queued builds on the controller before you can delete it. You can do this via the VSTS browser interface.

    Problem adding external AAD user to a directory backed VSTS instance

    Background

    I recently decided to change one of my VSTS instance to be directory backed. What this means is that in the past users logged in using LiveIDs (MSAs by their new name); once the VSTS instance was linked to an Azure Active Directory (AAD), via the Azure portal, they could login only if

  • they were using an account in the AAD
  • their MSA was listed as a guest in the AAD
  • they used a work ID in another AAD that is listed as a guest in my AAD
  • Thus giving me centralised user management.

    So I made the changes required, and the first two types of user were fine, but I had a problem with the third case. When I did the following

  • Added and external Work ID to my AAD directory (via the old management portal https://manage.windowsazure.com)
  • Added the user in my VSTS instance as a user
  • Granted the new user rights to access team projects.
  • All seemed to go OK, but when I tried to login as the user I got the error

    TF400813: The user 'db0990ce-80ce-44fc-bac9-ff2cce4720af\fez_blackmarble.com#EXT#@richardblackmarbleco.onmicrosoft.com' is not authorized to access this resource.

    clip_image002

    Solution

    With some help from Microsoft I got this fixed, seem to be an issue with Azure AD. The fix was to do the following

    1. Remove the user from VSTS account
    2. Go to the new Azure Portal (https://portal.azure.com/) and remove this user from the AAD
    3. Then re-add them as an external user back into the AAD (an invite email is sent)
    4. Add the user again to VSTS (another invite email is sent)
    5. Grant the user rights to the required team projects

    and this fixed the access problems for me. The key item for me I think was to use the new Azure portal.

    Hope it saves you some time

    There are now no excuses for not using Continuous Delivery from VSTS for Azure Web Apps

    One type of feature I hate people demoing in any IDE, especially Visual Studio, is the ‘just click here to publish to live from the developers PC’. This is just not good practice, we want to encourage a good DevOps process with

    • Source Control
    • Automated build
    • Automated release with approvals

    The problem is, this can all be a bit much for people, it takes work and knowledge, and that right click is just too tempting.

    So I was really pleased to see the new ‘Continuous Delivery (Preview)’ feature on Azure Web Apps announced at Connect().

    image

    This provides that one click simplicity, but creates a reasonably good DevOps pipeline using the features of VSTS using VSTS itself or GitHub as the source repository.

    For details of the exact features and how to use it see the ALM Blog post, I am sure it will provide you with a good starting point for your ongoing development if you don’t want to build it from scratch; but remember this will not be your end game, you are probably still going to need to think how you are going to manage the further config settings, tests and approvals a full process will require. It is just a much better place to start than a right click in Visual Studio.

    As announced at Connect() there is now a tool to fully migrate an on-premises TFS to VSTS

    I am often asked asked ‘How can I move my TFS installation to VSTS?’

    In the past the only real answer I  had was the consultant’s answer ‘it depends’. There were options, but they all ended up losing fidelity i.e. that the history of past changes got removed or altered in some manner. For many companies the implication of such changes meant they stayed on-premises; with all the regular backups, updates and patch running the use of any on-premises service entails.

    This has all changed with the announcement of the public preview of the TFS to VSTS Migrator from Microsoft at the Connect() conference.

    image

    In essence this allows a TFS Team Project Collection to be imported into VSTS as new VSTS instance. This makes it sound simple, but this can be a complex process depending upon your adoption of Azure Active Directory, the levels of customisation that have been made to your on-premises TFS instance and may require the upgrading your TFS server to the current version. Hence, the process is Microsoft ALM/DevOps partner led, and I am pleased to say that Black Marble is one of those Gold Partners.

    So if you have an on-premise TFS and…

    • your company strategy is cloud first and you want to migrate, with full history
    • or you don’t want to patch your TFS server any more (or you stopped doing it a while ago)
    • or you just want to move to VSTS because it where all the cool new bits are

    why not get in touch with us at Black Marble or myself to help you investigate the options.

    Devils with Arms and Cats - the new name for DevOps?

    Great day at DDDNorth yesterday, hope everyone enjoyed it. Thanks to all the team who helped during the preparation and on the day.

    The slides from Rik Hepworth and my presentation on ‘Living the dream - Real world DevOps with Azure and VSTS’ are up at Github

    We were a late stand in session  to cover for a presenter who could not attend on the day. So I hope it was not too much of let down, that we were not the speaker on the agenda, covered a different subject and did not match the title the spell checker converted out session title too. Though ‘Devils with Arms and Cats’  is maybe a good term for DevOps?

    As to the grok talk I did at lunchtime on developing VSTS extension with VS Code, there are no slides; but look at these past posts, building VSTS tasks with Powershell and putting a release process around vsts extension development they are on similar subjects.

    If I add a custom field to a VSTS work item type what is it’s name?

    The process customisation options in VSTS are now fairly extensive. You can add fields, states and custom items, making VSTS is ‘very possible’ option for many more people.

    As well as the obvious uses of this customisation such as storing more data or matching your required process, customisation can also aid in migrating work items into VSTS from other VSTS instances, or on-premises TFS.

    Whether using TFS Integration (now with no support – beware) or Martin Hinshelwood’s vsts-data-bulk-editor (an active open source solution so probably a much better choice for most people) as mentioned in my past post you need to add a custom field on the target VSTS server to contain the original work item ID. Commonly called ReflectedWorkItemId

    This can be added in VSTS  add detailed in MSDN

     

    image

    Note: In the case of Martin’s tool the field needs to be a string as it is going to contains a URL not the simple Integer you might expect.

    The small issue you have when you add a custom field is that this UI does not make it clear what the full name of field is. You need to remember that it is in the form <name of custom process>.<field name> e.g.  MigrateScrum.ReflectedWorkItemId.

    If you forget this you can always download the work item definition using the TFS Power Tools to have a look (yes this even works on VSTS).

    image