More examples of using custom variables in Azure DevOps multi-stage YML

I have blogged in the past ( here , here and here) about the complexities and possible areas of confusion with different types of Azure DevOps pipeline variables.

Well here is another example of how to use variables and what can trip you up.

The key in this example is the scope of a variable, whether it is available outside a job and the syntax to access it

Variables local to the Job

So, if you create your variable as shown below

write-host "##vso[task.setvariable variable=standardvar]$MyPowerShellVar"

It is only available in the current job in the form $(standardvar)

Variable with a wider scope

If you want it to be available in another job, or stage you have to declare it thus, adding ;isOutput=true

write-host "##vso[task.setvariable variable=stagevar;isOutput=true]$MyPowerShellVar"

But there is also a change in how you access it.

  • You need to give the script that declares the variable a name so it can be referenced
  • You need to add dependons associations between stages/jobs
  • And the syntax used to access the variable changes depending on whether you are in the same job, same stage but a different job or a completely different stage.

Below is a fully worked example

Updating the Azure Application client_secret used by Packer

As I have posted about previously, we create our Azure DevOps build agent images using the same Packer definitions as used by Microsoft. This time when I ran my Packer command to build an updated VHD I got the error

Build ‘vhd’ errored after 135 milliseconds 708 microseconds: adal: Refresh request failed. Status Code = ‘401’. Response body: {“error”:”invalid_client”,”error_description”:”AADSTS7000222: The provided client secret keys for app ‘6425416f-aa94-4c20-8395-XXXXXXX’ are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: 65a200cf-8423-4d52-af07-67bf26225200\r\nCorrelation ID: 0f86de87-33fa-443b-8186-4de3894972e1\r\nTimestamp: 2022-05-03 08:36:50Z”,”error_codes”:[7000222],”timestamp”:”2022-05-03 08:36:50Z”,”trace_id”:”65a200cf-8423-4d52-af07-67bf26225200″,”correlation_id”:”0f86de87-33fa-443b-8186-4de3894972e1″,”error_uri”:”https://login.microsoftonline.com/error?code=7000222″} Endpoint https://login.microsoftonline.com/545a7a95-3c4d-4e88-9890-baa86d5fdacb/oauth2/token==> Builds finished but no artifacts were created.

As the error message made clear, the client_secret had expired.

This value was originally set/generated when the Azure Service Prinicple was created. However, as I don’t want a new SP, this time I just wanted to update the secret via the Azure Portal (Home > AAD > Application Registration > [My Packer App].

The overview showed the old Secret had expired and I was able to create a new one on the Certificates and Secrets tab. However, when I update my Packer configuration file and re-ran the command it still failed.

It only worked after I deleted the expired secret. I am not sure if this is a requirement ( it is not something I have seen before) or just some propagation/cache delay.

But worth a blog post as a reminder to my future self and any other with a similar issue.

How to fix Azure Pipeline YAML parsing errors seen after renaming the default Git branch

If in Azure DevOps you rename your Git Repo’s default branch, say from ‘master’ to ‘main’, you will probably see an error in the form ‘Encountered error(s) while parsing pipeline YAML: Could not get the latest source version for repository BlackMarble.NET.App hosted on Azure Repos using ref refs/heads/master.‘ when you try to manually queue a pipeline run.

You could well think, as I did, ‘all I need to do is update the YAML build files with a find and replace for master to main’, but this does not fix the problem.

The issue is in the part of Azure DevOps pipeline settings that are still managed by the UI and not the YAML file. The association of the Git repo and branch. To edit this setting use the following process (and yes it is well hidden)

  • In the Azure DevOps browser UI open the pipeline for editing (it shows the YAML page)
  • On the ellipsise menu ( … top right) pick Tiggers
  • Select the YAML tab (on left)
  • Then select the ‘Get Sources’ section where you can change the default branch
  • Save the changes

Hope this post saves someone some time

Tidying up local branches with a Git Alias and a PowerShell Script

It is easy to get your local branches in Git out of sync with the upstream repository, leaving old dead branches locally that you can’t remember creating. You can use the prune option on your Git Fetch command to remove the remote branch references, but that command does nothing to remove local branches.

A good while ago, I wrote a small PowerShell script to wrapper the running of the Git Fetch and then based on the deletions remove any matching local branches. Then finally returning me to my trunk branch.

Note: This script was based on some sample I found, but I can’t remember where to give credit, sorry.

I used to just run this command from the command line, but I recently thought it would be easier if it became a Git Alias. As Git Aliases run a bash shell, this meant I needed to shell out to PowerShell 7. Hence, my Git Config ended up being as shown below

[user]
        name = Richard Fennell
        email = richard@blackmarble.co.uk
[filter "lfs"]
        required = true
        clean = git-lfs clean -- %f
        smudge = git-lfs smudge -- %f
        process = git-lfs filter-process
[init]
        defaultBranch = main
[alias]
        tidy = !pwsh.exe C:/Users/fez/OneDrive/Tools/Remove-DeletedGitBranches.ps1 -force

I can just run 'git tidy‘ and all my branches get sorted out.

How can I automatically create Azure DevOps Release Notes and how can I publish them?

A question I am often asked when consulting on Azure DevOps is ‘how can I automatically create release notes and how can I publish them?’.

Well it is for just this requirement that I have written a set of Azure DevOps Pipeline Tasks

  • Release Note Generator – to generate release notes. I strongly recommend this Cross-platform Node-based version. I plan to deprecate my older PowerShell version in the not too distant future as it uses ‘homegrown logic’, as opposed to standard Azure DevOps API calls, to get associated items.
  • Wiki Updater – to upload a page tot a WIKI.
  • WIKI PDF Generator – to convert a generated page, or whole WIKI, to PDF format.

So lets deal with these tools in turn

Generating Release Notes

The Release Note task generates release notes by getting the items associated with a build (or release) from the Azure DevOps API and generating a document based on a Handlebars based template.

  • The artefacts that can be included in the release notes are details of the build/release and associated Work Items, Commits/Changesets, Tests and Pull Requests.
  • Most of the sample templates provided are for markdown format files. However, they could easily be converted for other text-based formats such as HTML if needed.
  • The use of Handlebars are the templating language makes for a very flexible and easily extensible means of document generation. There are sample of custom extensions provided with the templates

Sample YAML for this task is as follows, not it is using an inline template but it is possible to also load the template from a file path

        - task: richardfennellBM.BM-VSTS-XplatGenerateReleaseNotes.XplatGenerate-Release-Notes.XplatGenerateReleaseNotes@3
          displayName: 'Generate Release Notes'
          inputs:
            outputfile: '$(System.DefaultWorkingDirectory)\inline.md'
            outputVariableName: OutputText
            templateLocation: InLine
            inlinetemplate: |
              # Notes for build 
              **Build Number**: {{buildDetails.id}}
              **Build Trigger PR Number**: {{lookup buildDetails.triggerInfo 'pr.number'}} 

              # Associated Pull Requests ({{pullRequests.length}})
              {{#forEach pullRequests}}
              {{#if isFirst}}### Associated Pull Requests (only shown if  PR) {{/if}}
              *  **PR {{this.id}}**  {{this.title}}
              {{/forEach}}

              # Builds with associated WI/CS ({{builds.length}})
              {{#forEach builds}}
              {{#if isFirst}}## Builds {{/if}}
              ##  Build {{this.build.buildNumber}}
              {{#forEach this.commits}}
              {{#if isFirst}}### Commits {{/if}}
              - CS {{this.id}}
              {{/forEach}}
              {{#forEach this.workitems}}
              {{#if isFirst}}### Workitems {{/if}}
              - WI {{this.id}}
              {{/forEach}} 
              {{/forEach}}

              # Global list of WI ({{workItems.length}})
              {{#forEach workItems}}
              {{#if isFirst}}## Associated Work Items (only shown if  WI) {{/if}}
              *  **{{this.id}}**  {{lookup this.fields 'System.Title'}}
                - **WIT** {{lookup this.fields 'System.WorkItemType'}} 
                - **Tags** {{lookup this.fields 'System.Tags'}}
              {{/forEach}}

              {{#forEach commits}}
              {{#if isFirst}}### Associated commits{{/if}}
              * ** ID{{this.id}}** 
                -  **Message:** {{this.message}}
                -  **Commited by:** {{this.author.displayName}} 
                -  **FileCount:** {{this.changes.length}} 
              {{#forEach this.changes}}
                    -  **File path (TFVC or TfsGit):** {{this.item.path}}  
                    -  **File filename (GitHub):** {{this.filename}}  
              {{/forEach}}
              {{/forEach}}

How to Publish The Notes

Once the document has been generated there is a need for a decision as to how to publish it. TThere are a few options

  • Attach the markdown file as an artefact to the Build or Pipeline. Note you can’t do this with a UI based Releases as they have no concept of artefacts, but this is becoming less of a concern as people move to multistage YAML.
  • Save in some other location e.g Azure Storage or if on-premises a UNC file share
  • Send the document as an email – I have used Rene van Osnabrugge Send Email Task for this job.
  • Upload it to a WIKI using my WIKI Updater Task
  • Convert the markdown release note document, or the whole WIKI, to a PDF and use any of the above options using first my WIKI PDF Exporter Task then another task.

I personally favour the 1st and 4th options used together. Attachment to the pipeline and then upload the document to a WIKI

A sample of suitable YAML is shown below, uploading the document to an Azure DevOps WIKI. Please note that the repo URL and authentication can trip you up here so have a good read of the provided documentation before you use this task.

  - task: richardfennellBM.BM-VSTS-WIKIUpdater-Tasks.WikiUpdaterTask.WikiUpdaterTask@1
          displayName: 'Git based WIKI Updater'
          inputs:
            repo: 'dev.azure.com/richardfennell/Git%20project/_git/Git-project.wiki'
            filename: 'xPlatReleaseNotes/build-Windows-handlebars.md'
            dataIsFile: true
            sourceFile: '$(System.DefaultWorkingDirectory)\inline.md'
            message: 'Update from Build'
            gitname: builduser
            gitemail: 'build@demo'
            useAgentToken: true

But when do I generate the release notes?

I would suggest you always generate release notes every build/pipeline i.e. a document of the changes since the last successful build/pipeline of that build definition. This should be attached as an artefact.

However, this per build document will usually too granular for use as ‘true’ release notes i.e. something to hand to a QA team, auditor or client.

To address this second use case I suggest, within a multistage YAML pipeline (or a UI based release), having a stage specifically for generating release notes.

My task has a feature that it will check for the last successful release of a pipeline/release to the stage it is defined in, so will base the release note on the last successful release to that given stage. If this ‘documentation’ stage is only run when you are doing a ‘formal’ release, the release note generated will be since the last formal release. Exactly what a QA team or auditor or client might want.

In conclusion

So I hope that this post provides some ideas as to how you can use my tasks generate some useful release notes.