BM-Bloggers

The blogs of Black Marble staff

New version of my Parameters.Xml Generator Visual Studio add-in now supports VS2017 too

I have just published Version 1.5 of my Parameters.Xml Generator Visual Studio add-in . After much fiddling this VSIX now supports VS2017 as well as VS2013 and VS2015.

The complexity was that VS2017 uses a new VSIX format, V3. You have to makes changes to the project that generates the VSIX and to the VSIX manifest too. The FAQ says you can do this within VS2015 my hand, but I had no luck getting it right. The recommended option, and the method I used, is to upgrade your solution to VS2017 (or the RC at the time of writing as the product has not RTM’d yet).

This upgrade process is a one way migration, and you do have to check/edit some key items

  • Get all your references right, as I was using the RC of VS2017  this meant enabling the use of Preview packages from Nuget in the solution.
  • Makes sure the install targets (of Visual Studio) match what you want to install too
  • Add prerequisites (this is the big new addition in the VSIX 3 format)
  • And the one that stalled me for ages – Make sure you reference the right version of the Microsoft.VisualStudio.Shell.<VERSION>.dll . You need to pick the one for the oldest version of Visual Studio you wish to target. In my case this was Microsoft.VisualStudio.Shell.12.0.dll. For some reason during the migration this got changed to Microsoft.VisualStudio.Shell.14.0.dll which gave the strange effect that the VSIX installed on 2013, 2015 and 2017. But in 2013, though I could see menu item, it did not work. This was fixed by referencing the 12.0 DLL .

Can’t add users to a VSTS instance backed by an Azure Directory

I have a VSTS instance that is backed by an Azure Directory. This is a great way to help secure a VSTS instance, only users in the Azure Directory can be added to VSTS, not just any old MSA (LiveIDs). This is a directory that can be shared with any other Azure based services such as O365, and centrally managed and linked to an on-premises Active Directory.

When I tried to add a user to VSTS, one that was a valid user in the Azure Directory, their account did not appear in the available users drop down.

 image

Turns out the problem was who I was logged in as. As yo can see from the screenshot I have three Richard accounts in the VSTS instance (and Azure Directory), a couple of MSAs and a guest work account from another Azure Directory. I was logged in as the guest work account.

All three IDs as administrators in VSTS, but it turned out I needed to be logged in as the MSA that owned the Azure subscription contains the Azure Directory. As soon as I used this account the dropdown populated as expected and I could add the users from the Azure Diretcory

image

Transform tool for transferring TFS 2015.3 Release Templates to VSTS

If you are moving from on-premises TFS to VSTS you might hit the same problem I have just have. The structure of a VSTS releases is changing, there is now the concept of multiple ‘Deployment Steps’ in an environment. This means you can use a number of different agents for a single environment – a good thing.

The downside this that if you export a TFS2015.3 release process and try to import it to VSTS it will fail saying the JSON format is incorrect.

Of course you can get around this with some copy typing, but I am lazy, so….

I have written a quick transform tool that converts the basic structure of the JSON to the new format. You can see the code as Github Gist

It is a command line tool, usage is as follows

  1. In VSTS create a new empty release, and save it
  2. Use the drop down menu on the newly saved release in the release explorer and export the file. This is the template for the new format e.g. template.json
  3. On your old TFS system export the release process in the same way to get your source file e.g. source.json
  4. Run the command line tool providing the name of the template, source and output file

    RMTransform template.json source.json output.json
  5. On VSTS import the newly create JSON file release file.
  6. A release process should be created, but it won’t be possible to save it until you have fixed a few things that are not transferred
    1. Associated each Deployment step with Agent Pool
    2. Set the user accounts who will do the pre-and post approvals
    3. Any secret variable will need to be reentered
      IMPORTANT - Make sure you save the imported process as soon as you can (i.e. straight after fixing anything that is stopping it being saved). If you don't save and start clicking into artifacts or global variable it seems to loose everything and you need to re-import

image

It is not perfect, you might find other issues that need fixing, but it save a load of copy typing

Deleting unwanted orphan XAML Build Controllers on a migrated VSTS instance

Whilst working with the VSTS Data Import Service I ended up migrating a TFS TPC up to VSTS that had an old XAML Build Controller defined. I did not need this XAML build controller, in fact I needed to remove it because it was using my free private build controller slot. Problem was I could not find a way to remove it via the VSTS (or Visual Studio Team Explorer) UI, and the VM that had been running the build controller was long gone.

The way I got rid of it in the end was the TFS C# API and a quick command line tool as shown below.

Note that you will need to delete any queued builds on the controller before you can delete it. You can do this via the VSTS browser interface.

Problem adding external AAD user to a directory backed VSTS instance

Background

I recently decided to change one of my VSTS instance to be directory backed. What this means is that in the past users logged in using LiveIDs (MSAs by their new name); once the VSTS instance was linked to an Azure Active Directory (AAD), via the Azure portal, they could login only if

  • they were using an account in the AAD
  • their MSA was listed as a guest in the AAD
  • they used a work ID in another AAD that is listed as a guest in my AAD
  • Thus giving me centralised user management.

    So I made the changes required, and the first two types of user were fine, but I had a problem with the third case. When I did the following

  • Added and external Work ID to my AAD directory (via the old management portal https://manage.windowsazure.com)
  • Added the user in my VSTS instance as a user
  • Granted the new user rights to access team projects.
  • All seemed to go OK, but when I tried to login as the user I got the error

    TF400813: The user 'db0990ce-80ce-44fc-bac9-ff2cce4720af\fez_blackmarble.com#EXT#@richardblackmarbleco.onmicrosoft.com' is not authorized to access this resource.

    clip_image002

    Solution

    With some help from Microsoft I got this fixed, seem to be an issue with Azure AD. The fix was to do the following

    1. Remove the user from VSTS account
    2. Go to the new Azure Portal (https://portal.azure.com/) and remove this user from the AAD
    3. Then re-add them as an external user back into the AAD (an invite email is sent)
    4. Add the user again to VSTS (another invite email is sent)
    5. Grant the user rights to the required team projects

    and this fixed the access problems for me. The key item for me I think was to use the new Azure portal.

    Hope it saves you some time

    Changes to LiveID/MSA and what I have done about it to get around the new domain limitations

    What are the changes in allowed email addresses in MSAs?

    You may or may not have noticed that there has been a recent change in LiveID (or Microsoft Account MSA as they are now called). In the past you could create a MSA using an existing email address e.g. richard@mydomain.co.uk . This is no longer an option. If you try to create a new MSA with a non Microsoft (outlook.com/hotmal.com) email they are blocked saying ‘This email is part of a reserved domain. Please enter a different email address’.

    image

    This limitation is actually a bit more complex than you might initially think, as it is not just your primary corporate work email it checks, it also checks any aliases you have. So in my case it would give the same error for richard@mydomain.com as well as richard@mydomain.co.uk because they are both valid non Microsoft domains even though one is really only used as an email alias for the other.

    So if creating a new MSA you will need to create a user@outlook.com style address. This is something all our new staff need to do as at present you need an MSA to associate with your MSDN subscription.

    In the past we asked them to create this MSA with their email user@mydomain.co.uk alias. This email address is an alias for their primary work email account, not their primary work account address user@mydomain.com itself. We encouraged them to not use their primary email address as it gets confusing as to which type of account (MSA or work account)  is in use at any given login screen if their login name is the same for both (their primary email address). We now we have to ask them to create one in the form bm-user@outlook.com to associate their MSDN subscription with.

    So that is all good, but that about any existing accounts?

    I think the best option is to update any existing to use new user@outlook.com addresses. I have found if you don’t do this you get into a place where the Azure/VSTS/O365 etc. login get confused as to whether your account is MSA or a Work Account. I actually managed to get to the point where I suddenly could not login to an Azure Active Directory (AAD) backed VSTS instance due to this confusion (the fix was to remove my ‘confused’ MSA and re-add my actual corporate AAD work account)

    How do I fix that then?

    To try to forestall this problem on other services I decided to update my old MSA email adress by do the following

    1. Login as my old MSA
    2. Go to https://account.microsoft.com
    3. Select ‘Info’
    4. Select ‘Manage how you sign in to Microsoft’
    5. Select ‘Add a new email address’
    6. Create a new @outlook.com email address (this will create a new email/Outlook inbox, but note that this seems to take a few minutes, or it did for me)
    7. Once the new email alias is created you can choose to make it your primary login address
    8. Finally you can delete your old address from the MSA

    And your are done, you now can login with your new user@outlook.com with your existing password and any 2FA settings  you have to any services you would previously login to e.g MSDN web site, VSTS etc.

    The one extra step I did was to go into https://outlook.live.com , one the new email inbox was created, to access the new Inbox and setup an email forward to my old richard@mydomain.co.uk email address. This was just to make sure any email notifications sent to the MSAs new Inbox end up somewhere I will actually see them, last think I wanted was a new Inbox to monitor

    Summary

    So I have migrated the primary email address for my MSA and all is good. You might not need this today, but I suspect it is something most people with MSAs using a work email as their primary login address are going to have to address at some point in the future.

    Can’t login to OneDrive desktop application on Windows 10

    Whilst I have been on holiday my PC has been switched off and in a laptop bag. This did not seem to stopped me getting problems when I tried to use it again…

    • Outlook could not sync to O365 – turns out there had been some changes in our hybrid Exchange infrastructure, I just need to restart/patch the PC on our company LAN to pick up all the new group policy settings etc.
    • Could not login to OneDrive getting a script error https://auth.gfx.ms/16.000.26657.00/DefaultLogin_Core.js

    image

    This second problem was a bit more complex to fix.

    1. Load Internet Explorer (not Edge)
    2. Go to Settings > Internet Options > Security
    3. Pick Trusted Sites and manually add the URL https://auth.gfx.ms as a trusted site.
    4. Unload the OneDrive desktop client
    5. Reload the OneDrive desktop client and you should get the usual LiveID login and all is good
    6. Interestingly – if you remove the trusted site setting the login still appears to work, but for how long I don’t know. I assume something is being cached.

    So it appears there have been a few changes on security whilst I have been away.

    Running Test Suites within a network Isolated Lab Management environment when using TFS vNext build and release tooling

    Updated 27 Sep 2016: Added solutions to known issues

    Background

    As I have posted many times we make use of TFS Lab Management to provide network isolated dev/test environments. Going forward I see us moving to Azure Dev Labs and/or Azure Stack with ARM templates, but that isn’t going to help me today, especially when I have already made the investment in setting up a Lab Management environments and they are ready to use.

    One change we are making now is a move from the old TFS Release Management (2013 generation) to the new VSTS and TFS 2015.2 vNext Release tools. This means I need to be able to trigger automated tests on VMs within Lab Management network isolated environments with a command inside my new build/release process. I have posted on how to do this with the older generation Release Management tools, turns out it is in some ways a little simpler with the newer tooling, no need to fiddle with shadow accounts etal.

    My Setup

    image

    Constraints

    The constraints are these

  • I need to be able to trigger tests on the Client VM in the network isolated lab environment. These tests are all defined in automated test suites within Microsoft Test Manager.
  • The network isolated lab already has a TFS Test Agent deployed on all the VMs in the environment linked back to the TFS Test Controller on my corporate domain, these agents are automatically installed and managed, and are handling the ‘magic’ for the network isolation – we can’t fiddle with these without breaking the Labs 
  • The new build/release tools assume that you will auto deploy a 2015 generation Test Agent via a build task as part of the build/release process. This is a new test agent install, so removed any already installed Test Agent – we don’t want this as it breaks the existing agent/network isolation.
  • So my only options to trigger the tests by using TCM (as we did in the past) from some machine in the system. In the past (with the old tools) this had to be within the isolated network environment due to the limitation put in place by the use of shadow accounts.  
  • However, TCM (as shipped with VS 2015) does not ‘understand’ vNext builds, so it can’t seem to find them by definition name/number – we have to find builds by their drop location, and I think this needs to be a UNC share, not a drop back onto the TFS server. So using TCM.EXE (and any wrapper scripts) probably is not going to deliver what I want i.e. the test run associated with a vNext build and/or release.
  • My Solution

    The solution I adopted was to write a PowerShell script that performs the same function as the TCMEXEC.PS1 script that used to be run within the network isolated Labe Environment by the older Release Management products.

    The difference is the old script shelled out to run TCM.EXE, my new version makes calls to the new TFS REST API (and unfortunately also to the older C# API as some features notably those for Lab Management services are not exposed via REST). This script can be run from anywhere, I chose to run it on the TFS vNext build agent, as this is easiest and this machine already had Visual Studio installed so had the TFS C# API available.

    You can find this script on my VSTSPowerShell GitHub Repo.

    The usage of the script is

    TCMReplacement.ps1
          -Collectionuri http://tfsserver.domain.com:8080/tfs/defaultcollection/
    -Teamproject "My Project"
    -testplanname "My test plan" 
    -testsuitename "Automated tests"
    -configurationname "Windows 8"
    -buildid  12345
       -environmentName "Lab V.2.0" 
    -testsettingsname "Test Setting"
    -testrunname "Smoke Tests"
    -testcontroller "mytestcontroller.domain.com"
    -releaseUri "vstfs:///ReleaseManagement/Release/167"
    -releaseenvironmenturi "vstfs:///ReleaseManagement/Environment/247"

    Note

  • The last two parameters are optional, all the others are required. If the last two are not used the test results will not be associated with a release
  • The is also a pollinginterval parameter which default to 10 seconds. The script starts a test run then polls on this interval to see if it has completed.
  • If there are any failed test then the script writes to write-error as the TFS build process sees this is a failed step
  • In some ways I think this script is an improvement over the TCMEXEC script, the old one needed you to know the IDs for many of the settings (loads of poking around in Microsoft Test Manager to find them), I allow the common names of settings to be passed in which I then use to lookup the required values via the APIs (this is where I needed to use the older C# API as I could not find a way to get the Configuration ID, Environment ID or Test Settings ID via REST).

    There is nothing stopping you running this script from the command line, but I think it is more likely to make it part of release pipeline using the PowerShell on local machine task in the build system. When used this way you can get many of the parameters from environment variables. So the command arguments become something like the following (and of course you can make all the string values build variables too if you want)

     

       -Collectionuri $(SYSTEM.TEAMFOUNDATIONCOLLECTIONURI) 
    -Teamproject $(SYSTEM.TEAMPROJECT)
    -testplanname "My test plan"
       -testsuitename "Automated tests"
    -configurationname "Windows 8"
    -buildid  $(BUILD.BUILDID)
      -environmentName "Lab V.2.0"
       -testsettingsname "Test Settings"
    -testrunname "Smoke Tests"
    -testcontroller "mytestcontroller.domain.com"
    -releaseUri $(RELEASE.RELEASEURI)
    -releaseenvironmenturi $(RELEASE.ENVIRONMENTURI)

     

    Obviously this script is potentially a good candidate for a TFS build/release task, but as per my usual practice I will make sure I am happy with it’s operation before wrappering it up into an extension.

    Known Issues

  • If you run the script from the command line targeting a completed build and release the tests run and are shown in the release report as well as on the test tab as we would expect.

    image

    However, if you trigger the test run from within a release pipeline, the test runs OK and you can see the results in the test tab (and MTM), but they are not associated within the release. My guess is because the release had not completed when the data update is made. I am investigating this to try to address the issue.
  • Previously I reported there was a known issue that the test results were associated with the build, but not the release. It turns out this was due to the AD account the build/release agent was running as was missing rights on the TFS server. To fix the problem I made sure the account as configured as follows”":

    Once this was done all the test results appeared where they should

    So hopefully you will find this a useful tool if you are using network isolated environments and TFS build

    If I add a custom field to a VSTS work item type what is it’s name?

    The process customisation options in VSTS are now fairly extensive. You can add fields, states and custom items, making VSTS is ‘very possible’ option for many more people.

    As well as the obvious uses of this customisation such as storing more data or matching your required process, customisation can also aid in migrating work items into VSTS from other VSTS instances, or on-premises TFS.

    Whether using TFS Integration (now with no support – beware) or Martin Hinshelwood’s vsts-data-bulk-editor (an active open source solution so probably a much better choice for most people) as mentioned in my past post you need to add a custom field on the target VSTS server to contain the original work item ID. Commonly called ReflectedWorkItemId

    This can be added in VSTS  add detailed in MSDN

     

    image

    Note: In the case of Martin’s tool the field needs to be a string as it is going to contains a URL not the simple Integer you might expect.

    The small issue you have when you add a custom field is that this UI does not make it clear what the full name of field is. You need to remember that it is in the form <name of custom process>.<field name> e.g.  MigrateScrum.ReflectedWorkItemId.

    If you forget this you can always download the work item definition using the TFS Power Tools to have a look (yes this even works on VSTS).

    image

    Fix for my Docker image create dates being 8 hours in the past

    I have been having a look at Docker for Windows recently. I have been experiencing a problem that when I create a new image the created date/time (as shown with docker images) is 8 hours in the past.

    image

    Turns out the problem seems to be due to putting my Windows 10 laptop into sleep mode. So the process to see the problem is

    1. Create a new Docker image – the create date is correct, the current time
    2. Sleep the PC
    3. Wake up the PC
    4. Check the create date, it is now 8 hours off in the past

    Now the create date is not an issue in itself, but the fact that the time within the Docker images is also off by 8 hours can be, especially when trying to connect to cloud based services. I needed to sort it out

    Turns out the fix is simple, you need to stop and restart the Docker process (or restarting the PC has the same effect as this restarts the Docker process). Why the Docker process ends up 8 hours off, irrespective of the time the PC is slept, I don’t know. Just happy to have a quick fix.