But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Problem adding external AAD user to a directory backed VSTS instance

Background

I recently decided to change one of my VSTS instance to be directory backed. What this means is that in the past users logged in using LiveIDs (MSAs by their new name); once the VSTS instance was linked to an Azure Active Directory (AAD), via the Azure portal, they could login only if

  • they were using an account in the AAD
  • their MSA was listed as a guest in the AAD
  • they used a work ID in another AAD that is listed as a guest in my AAD
  • Thus giving me centralised user management.

    So I made the changes required, and the first two types of user were fine, but I had a problem with the third case. When I did the following

  • Added and external Work ID to my AAD directory (via the old management portal https://manage.windowsazure.com)
  • Added the user in my VSTS instance as a user
  • Granted the new user rights to access team projects.
  • All seemed to go OK, but when I tried to login as the user I got the error

    TF400813: The user 'db0990ce-80ce-44fc-bac9-ff2cce4720af\fez_blackmarble.com#EXT#@richardblackmarbleco.onmicrosoft.com' is not authorized to access this resource.

    clip_image002

    Solution

    With some help from Microsoft I got this fixed, seem to be an issue with Azure AD. The fix was to do the following

    1. Remove the user from VSTS account
    2. Go to the new Azure Portal (https://portal.azure.com/) and remove this user from the AAD
    3. Then re-add them as an external user back into the AAD (an invite email is sent)
    4. Add the user again to VSTS (another invite email is sent)
    5. Grant the user rights to the required team projects

    and this fixed the access problems for me. The key item for me I think was to use the new Azure portal.

    Hope it saves you some time

    There are now no excuses for not using Continuous Delivery from VSTS for Azure Web Apps

    One type of feature I hate people demoing in any IDE, especially Visual Studio, is the ‘just click here to publish to live from the developers PC’. This is just not good practice, we want to encourage a good DevOps process with

    • Source Control
    • Automated build
    • Automated release with approvals

    The problem is, this can all be a bit much for people, it takes work and knowledge, and that right click is just too tempting.

    So I was really pleased to see the new ‘Continuous Delivery (Preview)’ feature on Azure Web Apps announced at Connect().

    image

    This provides that one click simplicity, but creates a reasonably good DevOps pipeline using the features of VSTS using VSTS itself or GitHub as the source repository.

    For details of the exact features and how to use it see the ALM Blog post, I am sure it will provide you with a good starting point for your ongoing development if you don’t want to build it from scratch; but remember this will not be your end game, you are probably still going to need to think how you are going to manage the further config settings, tests and approvals a full process will require. It is just a much better place to start than a right click in Visual Studio.

    As announced at Connect() there is now a tool to fully migrate an on-premises TFS to VSTS

    I am often asked asked ‘How can I move my TFS installation to VSTS?’

    In the past the only real answer I  had was the consultant’s answer ‘it depends’. There were options, but they all ended up losing fidelity i.e. that the history of past changes got removed or altered in some manner. For many companies the implication of such changes meant they stayed on-premises; with all the regular backups, updates and patch running the use of any on-premises service entails.

    This has all changed with the announcement of the public preview of the TFS to VSTS Migrator from Microsoft at the Connect() conference.

    image

    In essence this allows a TFS Team Project Collection to be imported into VSTS as new VSTS instance. This makes it sound simple, but this can be a complex process depending upon your adoption of Azure Active Directory, the levels of customisation that have been made to your on-premises TFS instance and may require the upgrading your TFS server to the current version. Hence, the process is Microsoft ALM/DevOps partner led, and I am pleased to say that Black Marble is one of those Gold Partners.

    So if you have an on-premise TFS and…

    • your company strategy is cloud first and you want to migrate, with full history
    • or you don’t want to patch your TFS server any more (or you stopped doing it a while ago)
    • or you just want to move to VSTS because it where all the cool new bits are

    why not get in touch with us at Black Marble or myself to help you investigate the options.

    Devils with Arms and Cats - the new name for DevOps?

    Great day at DDDNorth yesterday, hope everyone enjoyed it. Thanks to all the team who helped during the preparation and on the day.

    The slides from Rik Hepworth and my presentation on ‘Living the dream - Real world DevOps with Azure and VSTS’ are up at Github

    We were a late stand in session  to cover for a presenter who could not attend on the day. So I hope it was not too much of let down, that we were not the speaker on the agenda, covered a different subject and did not match the title the spell checker converted out session title too. Though ‘Devils with Arms and Cats’  is maybe a good term for DevOps?

    As to the grok talk I did at lunchtime on developing VSTS extension with VS Code, there are no slides; but look at these past posts, building VSTS tasks with Powershell and putting a release process around vsts extension development they are on similar subjects.

    If I add a custom field to a VSTS work item type what is it’s name?

    The process customisation options in VSTS are now fairly extensive. You can add fields, states and custom items, making VSTS is ‘very possible’ option for many more people.

    As well as the obvious uses of this customisation such as storing more data or matching your required process, customisation can also aid in migrating work items into VSTS from other VSTS instances, or on-premises TFS.

    Whether using TFS Integration (now with no support – beware) or Martin Hinshelwood’s vsts-data-bulk-editor (an active open source solution so probably a much better choice for most people) as mentioned in my past post you need to add a custom field on the target VSTS server to contain the original work item ID. Commonly called ReflectedWorkItemId

    This can be added in VSTS  add detailed in MSDN

     

    image

    Note: In the case of Martin’s tool the field needs to be a string as it is going to contains a URL not the simple Integer you might expect.

    The small issue you have when you add a custom field is that this UI does not make it clear what the full name of field is. You need to remember that it is in the form <name of custom process>.<field name> e.g.  MigrateScrum.ReflectedWorkItemId.

    If you forget this you can always download the work item definition using the TFS Power Tools to have a look (yes this even works on VSTS).

    image

    Typemock have released official VSTS build extension

    Typemock have just released an official VSTS build extension to run Typemock Isolator based tests. Given there is now an official extension I have decided to deprecate mine, it is still available in the Marketplace but I would recommend using the official one 

    The new Typemock extension includes two tasks

    SmartRunner Task

    The SmartRunner is a unit test runner, that can run nunit and mstest based tests. It handles the deployment of Typemock Isolator.  SmartRunner can run on both Shared and On Premises Agents

    Typemock with VSTests

    This task acts as a wrapper to enable Typemock Isolator and then run your tests via VSTest. This task can only be used with On Premises Agents as the build agent needs to be running with admin privileges.

    Why have I got a ‘.NETCore50’ and a ‘netcore50’ folder in my nuget package?

    I recently posted on how we were versioning our Nuget packages as part of a release pipeline. In test we noticed that the packages being produced by this process has an extra folder inside them.

    image 

    We expected there to be a netcore50 folder, but not a .NETCore50 folder. Strangely if we build the package locally we only saw the expect netcore50 folder. The addition of this folder did not appear to be causing any problem, but I did want to find out why it had appeared and remove it as it was not needed.

    Turns out the issue was the version of Nuget.exe, the automatically installed version on the on-prem TFS build agent was 3.2, my local copy 3.4. As soon as I upgraded the build box’s nuget.exe version to 3.4 the problem went away

    Experiences versioning related sets of NuGet packages within a VSTS build

    Background

    We are currently packaging up a set of UX libraries as NuGet packages to go on our internal NuGet server. The assemblies that make up the core of this framework are all in a single Visual Studio solution, however it makes sense to distribute them as a set of NuGet packages as you might not need all the parts in a given project. Hence we have a package structure as follows…

    • BM.UX.Common
    • BM.UX.Controls
    • BM.UX.Behaviours
    • etc…

    There has been much thought on the versioning strategy of these packages. We did consider independent versioning of each of these fundamental packages, but decided it was worth the effort, keeping their versions in sync was reasonable  i.e. the packages have the same version number and are released as a set.

    Now this might not be the case for future ‘extension’ packages, but it is an OK assumption for now, especially as it makes the development cycle quicker/easier. This framework is young and rapidly changing, there are often changes in a control that needs associated changes in the common assembly; it is hence good that a developers does not have to check-in a change on the common package before they can make an associated changed to the control package whist debugging a control prior to it being released.

    However, this all meant it was important to make sure the package dependencies and versions are set correctly.

    Builds

    We are using Git for this project (though this process is just as relevant for TFVC) with a development branch and a master branch. Each branch has its own CI triggered build

    • Development branch build …
      • Builds the solution
      • Runs Unit tests
      • Does SonarQube analysis
      • DOES NOT store any built artifacts
      • [Is used to validate Pull requests]
    • Master branch build …
      • Versions the code
      • Builds the solution
      • Runs Unit tests
      • Creates the NuGet Packages
      • Stores the created packages (to be picked up by a Release pipeline for publishing to our internal NuGet server)

    Versioning

    So within the Master build we need to do some versioning, this needs to be done to different files to make sure the assemblies and the NuGet packages are ‘stamped’ with the build version.

    We get this version for the build number variable, $(Build.BuildNumber), we use the format $(Major).$(Minor).$(Year:yy)$(DayOfYear).$(rev:r)  e.g. 1.2.16123.3

    Where

    • $(Major) and $(Minor) build variables we manage (actually our release pipeline updates the $(Minor) on every successful release to production using a VSTS task)
    • $(Year:yy)$(DayOfYear) gives a date in the form 16123
    • and $(rev:r) is a count of builds on a given day

    We have chosen to use this number format to version both the assemblies and Nuget packages, if you have different plans, such as semantic versioning , you will need to modify this process a bit.

    Assemblies

    The assemblies themselves are easy to version, we just need to set the correct value in their assemblyinfo.cs or assemblyinfo.vb files. I used my Assembly versioning VSTS task to do this

    NuGet Packages

    The packages turn out to be a bit more complex. Using the standard NuGet Packager task there is a checkbox to say to use the build number as the version. This works just fine versioning the actual package, adding the –Version flag to the package command to override the value in the project .nuspec file. However it does not help with managing the versions of any dependant packages in the solution, and here is why. In our build …

    1. AssemblyInfo files updated
    2. The solution is built, so we have version stamped DLLs
    3. We package the first ‘common’ Nuget package (which has no dependencies on other projects in the solution) and it is versioned using the –version setting, not the value in it’s nuspec file.
    4. We package the ‘next’ Nuget package, the package picks up the version from the –version flag (as needed), but it also needs to add a dependency to a specific version of the ‘common’ package. We pass the –IncludeReferencedProjects  argument to make sure this occurs. However, Nuget.exe gets this version number from  the ‘common’ packages .nuspec file NOT the package actually built in the previous step. So we end up with a mismatch.

    The bottom line is we need to manage the version number in the .nuspec file of each package. So more custom VSTS extensions are needed.

    Initially I reused my Update XML file task, passing in some XPath to select the node to update, and this is a very valid approach if using semantic versioning as it is a very flexible way yo build the version number. However, in the end I added an extra task to my versioning VSTS extension for Nuget to make my build neater and consistent with my other versions steps.

    Once all the versioning was done I could create the packages. I ended up with a build process as shown below

    image

    A few notes about the NuGet packaging

    • Each project I wish to create a Nuget package for has a nuspec file of the same ‘root’ name in the same folder as the csproj eg. mypackage.csproj and mypackage.nuspec. This file contains all descriptions, copyright details etc.
    • I am building each package explicitly, I could use wildcards in the ‘Path/Pattern to nuspec files’ property, I choose not to at this time. This is down to the fact I don’t want to build all the solution’s package at this point in time.
    • IMPORTANT I am passing in the .csproj file names, not the .nuspec file names to the ‘Path/Pattern to nuspec files’ property. I found I had to do this else the   –IncludeReferencedProjects  was ignored. The Nuget documentation seems to suggest as long as the .csproj and .nuspec files have the same ‘root’ name then you could reference the .nuspec file but this was not my experience
    • I still set the flag to use the build version to version the package – this is not actually needed as the .nuspec file has already been update
    • I pass in the  –IncludeReferencedProjects  argument via the advanced parameters, to pick up the project dependancies.

    Summary

    So now I have a reliable way to make sure my NuGet packages have consistent version numbers 

    Tidy up those VSTS release pipelines with meta-tasks

    Do you have repeating blocks in your VSTS release pipelines?

    I certainly do. A common one is to run a set of functional test, so I need to repeatedly …

    1. Deploy some test files to a VM
    2. Deploy a test agent to the VM – IMPORTANT I had not realised you can only run one test run against this deployed agent. You need to redeploy it for the next run
    3. Run my tests
    4. … and repeat for next test type/configuration/test plan/DLL etc.

     

    In the past this lead to a lot of repeat tasks in my release pipeline, all very messy.

    Now in VSTS we have the option of  Meta-tasks, these allow tasks to be grouped into in-effect functions with their own properties.

     

    image

    In the above screen shot below you can see I use a meta-task ‘Run Tests’ that wrappers the four tasks shown below.

    image

    Much neater, but as you might expect with something new I have come across a few minor gotchas

    • You cannot order the list of properties for the meta-task
    • This is a problem as the first one is used to generate the instance name in the pipeline. No a major problem you can always edit it.
    • Meta-tasks properties are auto-detected from any variables used with in the meta-task tasks, the auto-detection mechanism is case sensitive, unless the rest of VSTS variable handling. So be careful to not end up with duplicates.

    That all said, I think this is big step forward in readability and reuse for release management

    New version of my generate release notes task–now with authentication options

    I have just released 1.4.7 of the release notes VSTS extension. This provides a new advanced options that allows you to switch the authentication model.

    The default remains the same i.e. use a personal access token provided by the server, but you have the option to enable use of the 'defaultcredentials' (via the advanced properties). If this is done the account the build agent is running as is used. Hopefully this should fix the 401 issues some people have been seeing when using the task with on-prem TFS.

    For most people the default PAT model should be fine