Further enriching the data available in my Azure DevOps Pipelines Cross Platform Release Notes Task

I recently post about Enriching the data available in my Azure DevOps Pipelines Cross Platform Release Notes Task by adding Pull Request information. Well, that first release was fairly limited only working for PR validation builds, so I have made more improvements and shipped a newer version.

The task now will, as well as checking for PR build trigger, try to associate the commits associated with a build/release pipeline to any completed PRs in the repo. This is done using the Last Merge Commit ID, and from my tests seems to work for the various types of PR e.g. squash, merge, rebased and semi-linear.

The resultant set of PRs are made available to the release notes template processor as an array. However, there is a difference between the existing arrays for Work Items and Commits and the new one for Pull requests. The new one is only available if you are using the new Handlebars based templating mode.

You would add a block in the general form….

{{#forEach pullRequests}}
{{#if isFirst}}### Associated Pull Requests (only shown if  PR) {{/if}}
*  **PR {{this.id}}**  {{this.title}}

The reason I have chosen to only support Handlebars is it make the development so much easier and provides a more flexible solution, given all the Handlebar helpers available. I think this might be the first tentative step towards deprecating my legacy templating solution in favour of only shipping Handlebars support.

Swapping my Azure DevOps Pipeline Extensions release process to use Multistage YAML pipelines

In the past I have documented the build and release process I use for my Azure DevOps Pipeline Extensions and also detailed how I have started to move the build phases to YAML.

Well now I consider that multistage YAML pipelines are mature enough to allow me to do my whole release pipeline in YAML, hence this post.


My pipeline performs a number of stages, you can find a sample pipeline here. Note that I have made every effort to extract variables into variable groups to aid reuse of the pipeline definition. I have added documentation as to where variable are stored and what they are used for.

The stages are as follows


The build phase does the following

  • Updates all the TASK.JSON files so that the help text has the correct version number
  • Calls a YAML template (build-Node-task) that performs all the tasks to transpile a TypeScript based task – if my extension contained multiple tasks this template would be called a number of time
    • Get NPM packages
    • Run Snyk to check for vulnerabilities – if any vulnerabilities are found the build fails
    • Lint and Transpile the TypeScript – if any issue are found the build fails
    • Run any Unit test and publish results – if any test fail the build fails
    • Package up the task (remove dev dependencies)
  • Download the TFX client
  • Package up the Extension VSIX package and publish as a pipeline artifact.


The private phase does the following

  • Using another YAML template (publish-extension) publish the extension to the Azure DevOps Marketplace, but with flags so it is private and only assessible to my account for testing
    • Download the TFX client
    • Publishes the Extension to the Marketplace

This phase is done as a deployment job and is linked to an environment,. However, there are no special approval requirements are set on this environment. This is because I am happy for the release to be done to the private instance assuming the build phase complete without error.


This is where the pipeline gets interesting. The test phase does the following

  • Runs any integration tests. These could be anything dependant on the extension being deployed. Unfortunately there is no option at present in multistage pipeline for a manual task to say ‘do the manual tests’, but you could simulate similar by sending an email or the like.

The clever bit here is that I don’t want this stage to run until the new private version of the extension has been published and is available; there can be a delay between TFX saying the extension is published and it being downloadable by an agent. This can cause a problem in that you think you are running tests against a different version of the extension to one you have. To get around this problem I have implemented a check on the environment this stage’s deployment job is linked to. This check runs an Azure Function to check the version of the extension in the Marketplace. This is exactly the same Azure Function I already used in my UI based pipelines to perform the same job.

The only issue here is that this Azure Function is used as an exit gate in my UI based pipelines; to not allow the pipeline to exit the private stage until the extension is publish. I cannot do this in a multistage YAML pipeline as environment checks are only done on entry to the environment. This means I have had to use an extra Test stage to associate the entry check with. This was setup as follows

  • Create a new environment
  • Click the ellipse (…) and pick ‘approvals and checks’
  • Add a new Azure Function check
  • Provide the details, documented in my previous post, to link to your Azure Function. Note that you can, in the ’control options’ section of the configuration, link to a variable group. This is a good place to store all the values, you need to provide
    • URL of the Azure Function
    • Key to us the function
    • The function header
    • The body – this one is interesting. You need to provide the build number and the GUID of a task in the extension for my Azure Function. It would be really good if both of these could be picked up from the pipeline trying to use the environment. This would allow a single ‘test’ environment to be created for use by all my extensions, in the same way there are only a single ‘private’ and ‘public’ environment. However, there is a problem, the build number is picked up OK, but as far as I can see I cannot access custom pipeline variables, so cannot get the task GUID I need dynamically. I assume this is because this environment entry check is run outside of the pipeline. The only solution  can find is to place the task GUID as a hard coded value in the check declaration (or I suppose in the variable group). The downside of this is it means I have to have an environment dedicated to each extension, each with a different task GUID. Not perfect, but not too much of a problem
    • In the Advanced check the check logic
    • In control options link to the variable group contain any variables used.


The documentation stage again uses a template (generate-wiki-docs) and does the following


The public stage is also a deployment job and linked to an environment. This environment has an approval set so I have to approve any release of the public version of the extension.

As well as doing the same as private stage this stage does the following


It took a bit of trial and error to get this going, but I think I have a good solution now. The fact that the bulk of the work is done using shared templates means I should get good reuse of the work I have done. I am sure I will be able to improve the template as time goes on but it is a good start

My Azure DevOps Pipeline is not triggering on a GitHub Pull request – fixed

I have recently hit a problem that some of my Azure DevOps YAML pipelines, that I use to build my Azure DevOps Pipeline Extensions, are not triggering on a new PR being created on GitHub.

I did not get to the bottom of why this is happening, but I found a fix.

  • Check and of make a note of any UI declared variables in your Azure DevOps YAML Pipeline that is not triggering
  • Delete the pipeline
  • Re-add the pipeline, linking to the YAML file hosted on GitHub. You might be asked to re-authorise the link between Azure DevOps Pipelines and GitHub.
  • Re-enter any variables that are declared via the Pipelines UI and save the changes

Your pipeline should start to be triggered again

Enriching the data available in my Azure DevOps Pipelines Cross Platform Release Notes Task

A common request for my Generate Release Notes Tasks is to enrich the data available beyond basic build, work item and commit/changeset details. I have resisted these requests as it felt like a never ending journey to start. However, I have now relented and added the option to see any pull request information available.

This feature is limited, you obviously have to be using artifacts that linked to a Git repo, and also the Git repo have to on an Azure DevOps hosted repository. This won’t meet everyone’s needs but it is a start.

What was already available

Turns out there was already a means to get a limited set of PR details from a build. You used the form

**Build Trigger PR Number**: ${buildDetails.triggerInfo['pr.number']}

or in handlebars format

**Build Trigger PR Number**: {{lookup buildDetails.triggerInfo 'pr.number'}} 

The improvements

That said I have improved the options. There is now a new `prDetails` object available to the template.

If you use the dump option


You can see the fields available

     "repository": {
         "id": "bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59",
         "name": "VSTSBuildTaskValidation",
         "url": "https://richardfennell.visualstudio.com/670b3a60-2021-47ab-a88b-d76ebd888a2f/_apis/git/repositories/bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59",
         "project": {
             "id": "670b3a60-2021-47ab-a88b-d76ebd888a2f",
             "name": "GitHub",
             "description": "A container for GitHub CI/CD processes",
             "url": "https://richardfennell.visualstudio.com/_apis/projects/670b3a60-2021-47ab-a88b-d76ebd888a2f",
             "state": "wellFormed",
             "revision": 411511726,
             "visibility": 2,
             "lastUpdateTime": "2019-10-10T20:35:51.85Z"
         "size": 9373557,
         "remoteUrl": "https://richardfennell.visualstudio.com/DefaultCollection/GitHub/_git/VSTSBuildTaskValidation",
         "sshUrl": "richardfennell@vs-ssh.visualstudio.com:v3/richardfennell/GitHub/VSTSBuildTaskValidation",
         "webUrl": "https://richardfennell.visualstudio.com/DefaultCollection/GitHub/_git/VSTSBuildTaskValidation"
     "pullRequestId": 4,
     "codeReviewId": 4,
     "status": 1,
     "createdBy": {
         "displayName": "Richard Fennell (Work MSA)",
         "url": "https://spsprodeus24.vssps.visualstudio.com/Ac0efb61e-a937-42a0-9658-649757d55d46/_apis/Identities/b1fce0e9-fbf4-4202-bc09-a290def3e98b",
         "_links": {
             "avatar": {
                 "href": "https://richardfennell.visualstudio.com/_apis/GraphProfile/MemberAvatars/aad.NzQzY2UyODUtN2Q0Ny03YjNkLTk0ZGUtN2Q0YjA1ZGE5NDdj"
         "id": "b1fce0e9-fbf4-4202-bc09-a290def3e98b",
         "uniqueName": "bm-richard.fennell@outlook.com",
         "imageUrl": "https://richardfennell.visualstudio.com/_api/_common/identityImage?id=b1fce0e9-fbf4-4202-bc09-a290def3e98b",
         "descriptor": "aad.NzQzY2UyODUtN2Q0Ny03YjNkLTk0ZGUtN2Q0YjA1ZGE5NDdj"
     "creationDate": "2020-04-04T10:44:59.566Z",
     "title": "Added test.txt",
     "description": "Added test.txt",
     "sourceRefName": "refs/heads/branch2",
     "targetRefName": "refs/heads/master",
     "mergeStatus": 3,
     "isDraft": false,
     "mergeId": "f76a6556-8b4f-44eb-945a-9350124f067b",
     "lastMergeSourceCommit": {
         "commitId": "f43fa4de163c3ee0b4f17b72a659eac0d307deb8",
         "url": "https://richardfennell.visualstudio.com/670b3a60-2021-47ab-a88b-d76ebd888a2f/_apis/git/repositories/bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59/commits/f43fa4de163c3ee0b4f17b72a659eac0d307deb8"
     "lastMergeTargetCommit": {
         "commitId": "829ab2326201c7a5d439771eef5a57f58f94897d",
         "url": "https://richardfennell.visualstudio.com/670b3a60-2021-47ab-a88b-d76ebd888a2f/_apis/git/repositories/bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59/commits/829ab2326201c7a5d439771eef5a57f58f94897d"
     "lastMergeCommit": {
         "commitId": "53f393cae4ee3b901bb69858c4ee86cc8b466d6f",
         "author": {
             "name": "Richard Fennell (Work MSA)",
             "email": "bm-richard.fennell@outlook.com",
             "date": "2020-04-04T10:44:59.000Z"
         "committer": {
             "name": "Richard Fennell (Work MSA)",
             "email": "bm-richard.fennell@outlook.com",
             "date": "2020-04-04T10:44:59.000Z"
         "comment": "Merge pull request 4 from branch2 into master",
         "url": "https://richardfennell.visualstudio.com/670b3a60-2021-47ab-a88b-d76ebd888a2f/_apis/git/repositories/bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59/commits/53f393cae4ee3b901bb69858c4ee86cc8b466d6f"
     "reviewers": [],
     "url": "https://richardfennell.visualstudio.com/670b3a60-2021-47ab-a88b-d76ebd888a2f/_apis/git/repositories/bebd0ae2-405d-4c0a-b9c5-36ea94c1bf59/pullRequests/4",
     "supportsIterations": true,
     "artifactId": "vstfs:///Git/PullRequestId/670b3a60-2021-47ab-a88b-d76ebd888a2f%2fbebd0ae2-405d-4c0a-b9c5-36ea94c1bf59%2f4"

In templates this new object could be is used

**PR Title **: ${prDetails.title}

or in handlebars format.

**PR Details**: {{prDetails.title}}

It will be interesting to here feedback from the real world as opposed to test harnesses

Experiences setting up Azure Active Directory single sign-on (SSO) integration with GitHub Enterprise


GitHub is a great system for individuals and OSS communities for both public and private project. However, corporate customers commonly want more control over their system than the standard GitHub offering. It is for this reason GitHub offers  GitHub Enterprise.

For most corporates, the essential feature that GitHub Enterprise offers is the use Single Sign On (SSO) i.e. allowing users to login to GitHub using their corporate directory accounts.

I wanted to see how easy this was to setup when you are using Azure Active Directory (AAD).

Luckily there is a step by step tutorial from Microsoft on how to set this up. Though, I would say that though detailed this tutorial has a strange structure in that it shows the default values not the correct values. Hence, the tutorial requires close reading, don’t just look at the pictures!

Even with close reading, I still hit a problem, all of my own making, as I went through this tutorial.

The Issue – a stray / in a URL

I entered all the AAD URLs and certs as instructed (or so I thought) by the tutorial into the Security page of GitHub Enterprise.

When I pressed the ‘Validate’ button in GitHub, to test the SSO settings, I got an error

‘The client has not listed any permissions for ‘AAD Graph’ in the requested permissions in the client’s application registration’

This sent me shown a rabbit hole looking at user permissions. That wasted a lot of time.

However, it turns out the issue was that I had a // in a URL when it should have been a  /. This was because I had made a cut and paste error when editing the tutorial’s sample URL and adding my organisation details.

Once I fixed this typo the validation worked, I was able to complete the setup and then I could to invite my AAD users to my GitHub Enterprise organisation.


So the summary is, if you follow the tutorial setting up SSO from AAD to GitHub Enterprise is easy enough to do, just be careful of over the detail.

A major new feature for my Cross-platform Release Notes Azure DevOps Pipelines Extension–Handlebars Templating Support

I recently got a very interesting PR for my Cross-platform Release Notes Azure DevOps Pipelines Extension from Kenneth Scott. He had added a new templating engine to the task, Handlebars.

Previous to this PR the templating in the task was done with a line by line evaluation of a template that used my own mark-up. This method worked but has limitations, mostly due to the line by line evaluation model.  With the Kenneth’s PR the option was added to write your templates in Handlebars, or stay with my previous templating engine.

Using Handlebars

If you use Handlebars, the template becomes something like

## Notes for release  {{releaseDetails.releaseDefinition.name}}    
**Release Number**  : {{releaseDetails.name}}
**Release completed** : {{releaseDetails.modifiedOn}}     
**Build Number**: {{buildDetails.id}}
**Compared Release Number**  : {{compareReleaseDetails.name}}    

### Associated Work Items ({{workItems.length}})
{{#each workItems}}
*  **{{this.id}}**  {{lookup this.fields 'System.Title'}}
   - **WIT** {{lookup this.fields 'System.WorkItemType'}} 
   - **Tags** {{lookup this.fields 'System.Tags'}}

### Associated commits ({{commits.length}})
{{#each commits}}
* ** ID{{this.id}}** 
   -  **Message:** {{this.message}}
   -  **Commited by:** {{this.author.displayName}} 

The whole template is evaluated by the Handlebars engine using its own mark-up to provide a means for looping across arrays and the like.

This seemed a great enhancement to the task. However, we soon realised that it could be better. Handlebars is extensible, so why not allow the extensibility to be used?

Using Handlebars Extensions

I have added extensibility in two ways. Firstly I have also added support for the common Handlebar-Helpers extensions, this added over 150 helpers. These are just accessed in a template as follows

## To confirm the handbars-helpers is work
The year is {{year}} 
We can capitalize "foo bar baz" {{capitalizeAll "foo bar baz"}}

I have also added the ability to provide a block of JavaScript as a task parameter is that is loaded as a custom Handlebars extension. So if you add the following block in the tasks customHandlebarsExtensionCode parameter.

module.exports = {foo: function () {return 'Returns foo';}};

You can access in the templates as

## To confirm our custom extension works
We can call our custom extension {{foo}}

It will be interesting to see how popular this alternative way of templating will be.

Where did all my test results go?


I recently tripped myself up whist adding SonarQube analysis to a rather complex Azure DevOps build.

The build has two VsTest steps, both were using the same folder for their test result files. When the first VsTest task ran it created the expected .TRX and .COVERAGE files and then published its results to Azure DevOps, but when the second VsTest task ran it over wrote this folder, deleting the files already present, before it generated and published it results.

This meant that the build itself had all the test results published, but when SonarQube looked for the files for analysis only the second set of test were present, so its analysis was incorrect.


The solution was easy, use different folders for each set of test results.

This gave me a build, the key items are shown below, where one VsTest step does not overwrite the previous results before they can be processed by any 3rd party tasks such as SonarQube.

- task: SonarSource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@4
   displayName: 'Prepare analysis on SonarQube'
     SonarQube: Sonarqube
     projectKey: 'Services'
     projectName: 'Services'
     projectVersion: '$(major).$(minor)'
     extraProperties: |
      # Additional properties that will be passed to the scanner,

… other build steps

- task: VSTest@2
   displayName: 'VsTest – Internal Services'
     testAssemblyVer2: |
     searchFolder: '$(System.DefaultWorkingDirectory)/src/Services'
     resultsFolder: '$(System.DefaultWorkingDirectory)\TestResultsServices'
     overrideTestrunParameters: '-DeploymentEnabled false'
     codeCoverageEnabled: true
     testRunTitle: 'Services Unit Tests'
     diagnosticsEnabled: True
   continueOnError: true

- task: VSTest@2
   displayName: 'VsTest - External'
     testAssemblyVer2: |
     searchFolder: '$(System.DefaultWorkingDirectory)/src/ExternalServices'
     resultsFolder: '$(System.DefaultWorkingDirectory)\TestResultsExternalServices'
     vsTestVersion: 15.0
     codeCoverageEnabled: true
     testRunTitle: 'External Services Unit Tests'
     diagnosticsEnabled: True
   continueOnError: true

- task: BlackMarble.CodeCoverage-Format-Convertor-Private.CodeCoverageFormatConvertor.CodeCoverage-Format-Convertor@1
   displayName: 'CodeCoverage Format Convertor'
     ProjectDirectory: '$(System.DefaultWorkingDirectory)'

- task: SonarSource.sonarqube.6D01813A-9589-4B15-8491-8164AEB38055.SonarQubeAnalyze@4
   displayName: 'Run Code Analysis'

- task: SonarSource.sonarqube.291ed61f-1ee4-45d3-b1b0-bf822d9095ef.SonarQubePublish@4
   displayName: 'Publish Quality Gate Result'

You need to pass a GitHub PAT to create Azure DevOps Agent Images using Packer

I wrote recently about Creating Hyper-V hosted Azure DevOps Private Agents based on the same VM images as used by Microsoft for their Hosted Agent.

As discussed in that post, using this model you will recreate your build agent VMs on a regular basis, as opposed to patching them. When I came to do this recently I found that the Packer image generation was failing with errors related to accessing packages.

Initially, I did not read the error message too closely and just assumed it was an intermittent issue as I had found you sometime get random timeouts with this process. However, when the problem did not go away after repeated retries I realised I had a more fundamental problem, so read the log properly!

Turns out the issue is you now have to pass a GitHub PAT token that has at least read access to the packages feed to allow Packer to authenticate with GitHub to read packages.

The process to create the required PAT is as follows

  1. In a browser login to GitHub
  2. Click your profile (top right)
  3. Select Settings
  4. Pick Developer Settings
  5. Pick Personal Access Tokens and create a new one that has read:packages enabled


Once created, this PAT needs to be passed into Packer. If using the settings JSON file this is just another variable

"client_id": "Azure Client ID",
"client_secret": "Client Secret",
"tenant_id": "Azure Tenant ID",
"subscription_id": "Azure Sub ID",
"object_id": "The object ID for the AAD SP",
"location": "Azure location to use",
"resource_group": "Name of resource group that contains Storage Account",
"storage_account": "Name of the storage account",
"ssh_password": A password",
"install_password": "A password",
"commit_url": "A url to to be save in a text file on the VHD, usually the URL if commit VHD based on",

"github_feed_token": "A PAT"


If you are running Packer within a build pipeline, as the other blog post discusses, then the PAT will be another build variable.

Once this change was made I was able to get Packer to run to completion, as expected.

Creating Hyper-V hosted Azure DevOps Private Agents based on the same VM images as used by Microsoft for their Hosted Agents


There are times when you need to run Private Azure DevOps agents as opposed to using one of the hosted ones provided by Microsoft. This could be for a variety of reasons, including needing to access resources inside your corporate network or needing to have a special hardware specification or set of software installed on the agent.


If using such private agents, you really need to have an easy way to provision them. This is so that all your agents are standardised and easily re-creatable. Firstly you don’t want build agents with software on them you can’t remember installing or patching. This is just another form of the “works on one developer’s machine but not another” problem. Also if you have the means to replace the agents very regularly and reliably you can avoid the need to patch them; you can just replace them with newer VMs created off latest patched base Operating System images and software releases.

Microsoft uses Packer to build the VM images into Azure Storage. Luckily, Microsoft have open sourced their build tooling process and configuration, you can find the resources on GitHub

A fellow MVP, Wouter de Kort, has done an excellent series of posts on how to use these Packer tools to build your own Azure hosted Private Agents.

I don’t propose to go over that again. In this post, I will discuss what needs to be done to use these tools to create private agents on your own Hyper-V hardware.

By this point you are probably thinking ‘could this be done with containers? They are designed to allow the easy provisioning of things like agents’.

Well, the answer is yes that is an option. Microsoft provides both container and VM based agents and have only recently split the repo to separate the container creation logic from the VM creation logic. The container logic remains in the old GitHub home. However, in this post I am focusing on VMs, so will be working against the new home for the VM logic.

Preparation – Getting Ready to run Packer

Copy the Microsoft Repo

Microsoft’s needs are not ours, we wanted to make some small changes to the way that Packer builds VMs. The key changes are:

  • We want to add some scripts to the repo to help automate our process.
  • We don’t, at this time, make much use of Docker, so don’t bother to pre-cache the Docker images in the agent. This speeds up the image generation and keeps the VMs VHD smaller.

The way we manage these changes is to import the Microsoft repo into our Azure DevOps Services instance. We can keep our copy up to date by setting an upstream remote reference and from time to time merging in Microsoft’s changes, but more on that later.

All our are changes are done on our own long living-branch, we PR any revisions we make into this long lived branch.

The aim is to not alter the main Microsoft Packer JSON definition as sorting out a three way merge if both theirs and our versions of the main JSON file are updated is harder than I like. Rather if we don’t want a feature installed we add ’return $true’ at the start of the PowerShell script that installs the feature, thus allowing Packer to call the script, but skip the actions in the script without the need to edit the controlling JSON file.

This way of working allows us to update the master branch from the upstream repo to get the Microsoft changes, and then to regularly rebase our changes onto the updated master.


A local test of Packer

It is a good idea to test out the Packer build from a development PC to make sure you have all the Azure settings correct. This is done using a command along the lines of

packer.exe" build -var-file="azurepackersettings.json"  -on-error=ask "Windows2016-Azure.json"

Where the ‘windows2016-azure.json’ is the Packer definition and the ‘azurepackersettings.json’ the user configurations containing the following values. See the Packer documentation for more details

"client_id": "Azure Client ID",
"client_secret": "Client Secret",
"tenant_id": "Azure Tenant ID",
"subscription_id": "Azure Sub ID",
"object_id": "The object ID for the AAD SP",
"location": "Azure location to use",
"resource_group": "Name of resource group that contains Storage Account",
"storage_account": "Name of the storage account",
"ssh_password": A password",
"install_password": "A password",
"commit_url": "A url to to be save in a text file on the VHD, usually the URL if commit VHD based on"

If all goes well you should end up with a SysPrep’d VHD in your storage account after a few hours.

Note: You might wonder why we don’t try to build the VM locally straight onto our Hyper-V infrastructure. Packer does have a Hyper-V ISO builder but I could not get it working. Firstly finding an up to date patched Operative System ISO is not that easy and I wanted to avoid having to run Windows Update as this really slows the creation process . Also the process kept stalling as it could not seem to get a WinRM session, when I looked this seemed to be something to do with Hyper-V Vnet switches. In the end, I decided it was easier just to build to Azure storage. This also had the advantage of requiring fewer changes to the Microsoft Packer definitions, so making keeping our branch up to date easier.

Pipeline Process – Preparation Stages

The key aim was to automate the updating of the build images. So we aimed to do all the work required inside an Azure DevOps multistage pipeline. How you might choose to implement such a pipeline will depend on your needs, but I suspect it will follow a similar flow to ours.

  1. Generate a Packer VHD
  2. Copy the VHD locally
  3. Create a new agent VM from the VHD
  4. Repeat step 3. a few times

There is a ‘what comes first the chicken or the egg’ question here. How do we create the agent to run the agent creation on?

In our case, we have a special manually created agent that is run on the Hyper-V host that the new agents will be created on. This has some special configuration which I will discuss further below.

Stage 1 – Update our repo

As the pipeline has a source of our copy of the repo (and targets our branch), the pipeline will automatically get the latest version of our Packer configuration source in our repo. However, there is a very good chance Microsoft will have updated their upstream repo. We could of course manually update our repo as mentioned above and we do do this from time to time. However, just to make sure we are up to date, the pipeline also does a fetch, merge and rebases our branch on its local copy. To do this it does the following

  1. Adds the Microsoft repo as an upstream remote
  2. Fetches the latest upstream/master changes and merges them onto origin/master
  3. Rebases our working branch onto the updated origin/master

Assuming this all works, and we have not messed up a commit so causing a 3-way merge that blocks the scripts, we should have all Microsoft’s latest settings e.g packages, base images etc. plus our customisation.

Stage 2 – Run Packer

Next, we need to run Packer to generate the VHD. Luckily there is a Packer extension in the Marketplace. This provides two tasks we use

  1. Installs the Packer executable
  2. Run Packer passing in all the values, stored securely as Azure DevOps pipeline variables, as used in the azurepackersettings.json file for a local test plus the details of an Azure subscription.

Being run within a pipeline has no effect on performance, so this stage is still slow, taking hours. However, once it is completed we don’t need to run it again so we have this stage set for conditional execution based on a pipeline variable so we can skip the step if it has already completed. Very useful for testing.

Stage 3 – Copy the VHD to a Local File Share

As we are building local private agents we need the VHD file stored locally i.e. copied down to a local UNC share. This is done with some PowerShell that runs the Azure CLI. It finds the newest VHD in the Azure Storage account and copies it locally, we do assume we are the only thing creating VHDs in the storage account and that the previous stage has just completed.

Again this is slow, it can take many hours depending on how fast your internet connection is. Once the VHD file is downloaded, we create a metadata file contains the name of profile it can be used with e.g. for a VS2017 or VS2019 agent and a calculated VHD file checksum, more details on both of these below.

Now again, as this stage is slow, and once it is completed we don’t need to run it again, we have conditional execution based on a second build variable so we can skip the step if it is not needed.

If all runs Ok, then at this point we have a local copy of a SysPre’d VHD. This can be considered the preparation phase over. These stages need to be completed only once for any given generation of an agent.

Pipeline Process – Deployment Stages

At this point we now have a SysPre’d VHD, but we don’t want to have to generate each agent by hand completing the post SysPrep mini setup and installing the Azure DevOps Agent.

To automate this configuration process we use Lability. This is a PowerShell tool that wrappers PowerShell’s Desired State Configuration (DSC). Our usage of Lability and the wrapper scripts we use are

discussed in this post by a colleague and fellow MVP Rik Hepworth. However, the short summary is that Lability allows you to create an ‘environment’ which can include one or more VMs. In our case, we have a single VM in our environment so the terms are interchangeable in this post.

Each VM in an environment is based on one or more master disk images. Each instance of a VM uses its own Hyper-V diff disk off their master disk, thus greatly reducing the disk space required. This is very useful when adding multiple virtually identical agent VMs to a single Hyper-V host.

A Lability environment allows us to have a definition of what a build VM is i.e. what is its base VHD image, how much memory does it have, are there any extra disks, how many of CPU cores does it have, this list goes on. Also, it allows us to install software, in our case the Azure DevOps agent.

All the Lability definitions are stored in a separate Git repo. We have to make sure the current Lability definitions are already installed along with the Lability tools on the Azure DevOps agent that will be running these stages of the deployment pipeline. We do this by hand  on our one ‘special agent’ but it could be automated.

Remember, in our case, this ‘special agent’ is actually domain-joined, unlike all the agents we are about to create, and running on the Hyper-V host where we will be deploying the new VMs. As it is domain joined it can get to the previously downloaded Sysprep’d VHD and metadata file on a network UNC share. We are not too worried over the ‘effort’ keeping the Lability definitions update as they very rarely change, all changes tend to be in the Packer generated base VHD.

It should be remembered that this part of the deployment is a repeatable process, but we don’t just want to keep registering more and more agents. Before we add a new generation agent we want to remove an old generation one. Hence, cycling old agents out of the system, keeping things tidy.

We have experimented with naming of Lability environments to make it easier to keep things tidy. Currently, we provide two parameters into our Lability configuration

  • Prefix – A short-code to identify the role of the agent we use e.g. ‘B’ for generic build agents and ‘BT’ for ones with the base features plus BizTalk
  • Index – This number is used for two jobs, the first is to identify the environment in a set of environments of the same Prefix. It is also used to work out which Hyper-V VNet the new environment should be attached to on the Hyper-V host. Lability automatically deals with the creation of these VNets if not present.

So for on our system, for example, a VM will end up with a name in the form B1BMAgent2019, this means

  • B – It is a generic agent
  • 1 – It is on the subnet, and is the first of the B group of agents
  • BMAgent2019 – It is based on our VS2019 VHD image

Note: Also when an Azure DevOps Agent is registered with Azure DevOps, we also append a random number, based on the time, to the end of the agent name in Azure DevOps. This allows two VMs with the same prefix and index, but on different Hyper-V hosts, to be registered at the same time, or to have multiple agents on the same VM. In reality, we have not used this feature. We have ended up using unique prefix and index across agent our estate with a single agent per VM. 

Stage 4 – Disable the old agent then remove it

The first deployment step is done with a PowerShell script. We check to see if there is an agent registered with the current Prefix and Index. If there is we disable it via the Azure DevOps Rest API. This will not stop the current build but will stop the agent picking up a new one when the current one completes.

Once the agent is disabled we keep polling it, via the API, until we see it go idle. Once the agent is idle we can use the Azure DevOps API to delete the agent’s registration on Azure DevOps.

Stage 5 – Remove the old Environment

Once the agent is no longer registered with Azure DevOps we can then remove the environment running the agent. This is a Lability command that we wrapper in PowerShell scripts

This completely removes the Hyper-V VM and its diff disks that store its data, a very tidy process.

Stage 6 – Update Lability Settings

I said previously that we rarely need to update the Lability definitions. There is one exception, that is the reference to the base VHD. We need to update this to point to the copy of the Packer generated SysPrep’d VHD on the local UNC file share.

We use another PowerShell script to handle this. It scans the UNC share for metadata files to find the one containing the request media type e.g. VS2017 or VS2019 (we only keep one of each type there). It then registers this VHD in Lability using the VHD file path and the previously calculated checksum. Lability uses the checksum to work out if the file has been updated.

Stage 7 – Deploy New Environment

So all we have to do at this point is request Lability to create a new environment based on the variable parameters we pass in i.e. environment definition, prefix and any override parameters (the Azure DevOps Agent configuration) into a wrapper script.

When this script is run, it triggers Lability to create a new VM using an environment configuration.

Lability’s first step is to create the VNet if not already present.

It then checks, using the checksum, if the base Sysprep’d VHD has been copied to the Hyper-V host. If it has not been copied it is done before continuing. This can take a while but is only done once.

Next, the environment (our agent VM) is created, firstly the VM settings are set e.g. CPU & Memory and then the Windows mini setup is handled by DSC. This sets the following

  • Administrator user Account and Password
  • Networking, here we have to rename an Ethernet adapter. We have seen the name of the first Ethernet change across different versions of the Packer image, so to make our lives easier we rename the primary adaptor connected to the VNet to a known value.
  • Swap Disk, set this allow the Operating System to manage this as the default on the Packer image is to use a dedicated drive D: which we don’t have.
  • Create a dedicated drive E: for the agent.
  • Download, install and configure the Azure DevOps agent

DSC handles any reboots required.

After a few minutes, you should see a new registered agent in the requested Azure DevOps Agent Pool.

Stage 8 – Add Capabilities

Our builds make use of Azure DevOps user capabilities to target them to the correct type of agent. We use an yet another PowerShell script that waits until the new agent been registered and then it adds our custom capabilities from a comma-separated string parameter.

A little tip here. A side effect of our Lability configuration is that all the agents have the same machine name. This can make finding the host they are on awkward, especially if you have a few Hyper-V hosts. So to address this problem we add a capability of the Hyper-V hosts name, this is purely to make finding the VM easier if we have to.

Stage  9 – Copy Capabilities

We have seen that some of the Azure DevOps tasks we use have demands that are not met by the System Capabilities. The classic is a task requiring a value for the capability ‘java’ or ‘jdk’ when  the one that is present on the agent is ‘JAVA_HOME’.

To address this, as opposed to adding our own capability which might not point to the correct location, is to copy an existing capability that has the correct value. Again this is done with a PowerShell script that takes as string parameter

So what do we end up with?

When all this completes we have a running private agent that has all the features of the Microsoft hosted ones. As Microsoft adds new functionality or patch their agent images, as long as we regenerate our Packer images, we get the same features.

At this point in time, we have chosen to add any extra software we require after the end of this process, as opposed to within it. In our case, this is basically either BizTalk 2013 and BizTalk 2016 on a single agent in our pool. Again we do this with a series of scripts, but manually run this time. We would like to fully automate the process, but BizTalk does not lend itself to easy installation automation. So, after a good bit of experimentation, we decided the best option for now, was to keep our basic build process as close to the Microsoft Packer images as possible to minimise merge issues, and worry about BizTalk later. As we only have one BizTalk 2013 and one 2016 agent the cost of manually finishing off was not too high.

Where do we go from here?

We now have a process that is automated end to end. However, it can be ‘a little brittle’, but as all the stages tidy up after themselves rerunning jobs is not an issue other than in the time cost.

We still have not decided on a final workflow for the replacement of agent. At this time we use manual approvals before deploying an agent. I am sure this will change as we allow this process to mature.

It is a good starting point.

Major enhancements to my Azure DevOps Cross Platform Release Notes Extension

Over the past few days I have published two major enhancements to my Azure DevOps Cross Platform Release Notes Extension.

Added Support for Builds

Prior to version 2.17.x this extension could only be used in Releases. This was because it used Release specific calls provided in the Microsoft API to work out the work items and changesets/commits associated with the Release. This is unlike my older PowerShell based Release Notes Extension which was initially developed for Builds and only later enhanced to  work in Releases, but achieved this using my own logic to iterate across Builds associated with Releases to work out the associations.

With the advent of YAML multistage Pipelines the difference between a Build and a Release is blurring, so I thought it high time to add Build support to my Cross Platform Release Notes Extension. Which it now does.

Adding Tag Filters

In the Cross Platform Release Notes Extension you have been able to filter the work items returned in a generated document for a good while, but the filter was limited to a logical AND

i.e. if the filter was


All work items matched would have to have both the TAG1 and TAG2 set

Since 2.18.x there is now the option of a logic AND or an OR.

  • @@WILOOP:TAG1:TAG2@@ matches work items that have all tags (legacy behaviour for backward compatibility)
  • @@WILOOP[ALL]:TAG1:TAG2@@ matches work items that have all tags
  • @@WILOOP[ANY]:TAG1:TAG2@@ matches work items that any of the tags

Update 5th Dec

In 2.19.x there is also the option to filter on any field in a work item as well as tags

  • @@WILOOP[ALL]:System.Title=This is a title:TAG 1@@

For more details see the extension’s WIKI page


My plan is to at some point deprecate my PowerShell based Release Notes Extension. I have updated the documentation for this older extension state as much and to recommend the use of the newer Cross Platform Release Notes Extension.

At this time there is now little that this older extension can do that cannot be done by my newer Cross Platform Release Notes Extension. Moving to it I think makes sense for everyone for the…

  • Cross platform support
  • Use of the same means to find the associated items as the Microsoft UI to avoid confusion
  • Enhanced work item filtering

Lets see of the new features and updated advisory documentation effect the tow extension relative download statistics