More examples of using custom variables in Azure DevOps multi-stage YML

I have blogged in the past ( here , here and here) about the complexities and possible areas of confusion with different types of Azure DevOps pipeline variables.

Well here is another example of how to use variables and what can trip you up.

The key in this example is the scope of a variable, whether it is available outside a job and the syntax to access it

Variables local to the Job

So, if you create your variable as shown below

write-host "##vso[task.setvariable variable=standardvar]$MyPowerShellVar"

It is only available in the current job in the form $(standardvar)

Variable with a wider scope

If you want it to be available in another job, or stage you have to declare it thus, adding ;isOutput=true

write-host "##vso[task.setvariable variable=stagevar;isOutput=true]$MyPowerShellVar"

But there is also a change in how you access it.

  • You need to give the script that declares the variable a name so it can be referenced
  • You need to add dependons associations between stages/jobs
  • And the syntax used to access the variable changes depending on whether you are in the same job, same stage but a different job or a completely different stage.

Below is a fully worked example

Updating the Azure Application client_secret used by Packer

As I have posted about previously, we create our Azure DevOps build agent images using the same Packer definitions as used by Microsoft. This time when I ran my Packer command to build an updated VHD I got the error

Build ‘vhd’ errored after 135 milliseconds 708 microseconds: adal: Refresh request failed. Status Code = ‘401’. Response body: {“error”:”invalid_client”,”error_description”:”AADSTS7000222: The provided client secret keys for app ‘6425416f-aa94-4c20-8395-XXXXXXX’ are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.rnTrace ID: 65a200cf-8423-4d52-af07-67bf26225200rnCorrelation ID: 0f86de87-33fa-443b-8186-4de3894972e1rnTimestamp: 2022-05-03 08:36:50Z”,”error_codes”:[7000222],”timestamp”:”2022-05-03 08:36:50Z”,”trace_id”:”65a200cf-8423-4d52-af07-67bf26225200″,”correlation_id”:”0f86de87-33fa-443b-8186-4de3894972e1″,”error_uri”:”https://login.microsoftonline.com/error?code=7000222″} Endpoint https://login.microsoftonline.com/545a7a95-3c4d-4e88-9890-baa86d5fdacb/oauth2/token==> Builds finished but no artifacts were created.

As the error message made clear, the client_secret had expired.

This value was originally set/generated when the Azure Service Prinicple was created. However, as I don’t want a new SP, this time I just wanted to update the secret via the Azure Portal (Home > AAD > Application Registration > [My Packer App].

The overview showed the old Secret had expired and I was able to create a new one on the Certificates and Secrets tab. However, when I update my Packer configuration file and re-ran the command it still failed.

It only worked after I deleted the expired secret. I am not sure if this is a requirement ( it is not something I have seen before) or just some propagation/cache delay.

But worth a blog post as a reminder to my future self and any other with a similar issue.

Fix for Azure DevOps deployment to an environment stuck in “Job is pending” state

Issue

I had an Azure DevOps YAML based pipeline that had been working but was now getting stuck with the message “Job is pending…” when trying to start a stage in which there is a deployment to an environment.

Looking at the logs and Azure DevOps UI it was not obvious what the issue was.

Solution

Turns out it was due to environment checks and approvals. There was a branch policy on the environment. This was set to only allow use of Azure DevOps Templates on a given branch. The edit that had been done to the YAML meant it was trying to extend a template in a branch that was not in the approved list.

As soon as the working branch was added to the proved list it all worked as expected.

So if you see “Job is pending” errors with no obvious reason, check the environment approvals and checks. Remember, any issues with these don’t show up in the build log

A workaround for not being able to access custom variables via stagedependencies if they are set in deployment jobs in Azure DevOps Pipelines

I have blogged in the past ( here and here) about the complexities and possible areas of confusion with different types of Azure DevOps pipeline variables. I have also seen issues raised over how to access custom variables across jobs and stages. Safe to say, this is an area where it is really easy to get it wrong and end up with a null value.

I have recently come across another edge case to add to the list of gotchas.

It seems you cannot use stagedependencies to access a variable declared in a deployment job i.e. when you are using an environment to get approval for a release.

The workaround is to add a job that is dependent on the deployment and set the custom variable within it. This variable can be accessed by a later stage as shown below

- stage: S1
  jobs:
  - deployment: D1
    strategy:
      runOnce:
        deploy:
          steps:
              - checkout: none
              - bash: echo "Can't access the variable if set in here"
  - job: J1
    dependsOn:
      D1
    steps:
      - checkout: none
      - bash: echo "##vso[task.setvariable variable=myvar;isOutput=true]True" 
        name: BashStep

- stage: S2
  condition: always()

  dependsOn: 
   - S1
  jobs:
   - job: Use_Variable
     variables: # add an alias for the var
       myvar: $[stagedependencies.S1.J1.outputs['BashStep.myvar']]
        steps:
          - checkout: none
          - dash: echo "Script gets run when myvar is true"
            condition: eq (variables['myvar'],'True')

The importance of blogging – or how to do your future self a favour

Yesterday, yet again, I was thankful for my past self taking time to blog about a technical solution I had found.

I had an error when trying to digitally sign a package. On searching on the error code I came across my own blog post with the solution. This was, as usual, one I had no recollection of writing.

I find this happens all the time. It is a little disturbing when you search for an issue and the only reference is to a post you made and have forgotten, so you are the defacto expert, nobody knows anymore on the subject, but better than having no solution.

Too often I ask people if they have documented the hints, tips and solutions they find and the response I get is ‘I will remember’. Trust me you won’t. Write something down where it is discoverable for your team and your future self. This can be any format that works for you: an Email, OneNote, a Wiki or the one I find most useful a blog. Just make sure it is easily searchable.

Your future self will thank you.

Using Azure DevOps Stage Dependency Variables with Conditional Stage and Job Execution

I have been doing some work with Azure DevOps multi-stage YAML pipelines using stage dependency variables and conditions. They can get confusing quickly, you need one syntax in one place and another elsewhere.

So, here are a few things I have learnt…

What are stage dependency variables?

Stage Dependencies are the way you define which stage follows another in a multi-stage YAML pipeline. This is as opposed to just relying on the order they appear in the YAML file, the default order. Hence, they are critical to creating complex pipelines.

Stage Dependency variables are the way you can pass variables from one stage to another. Special handling is required, as you can’t just use the ordinary output variables (which are in effect environment variables on the agent) as you might within a job as there is no guarantee the stages and jobs are running on the same agent.

For stage dependency variables, is not how you create output variables, that does not differ from the standard manner, the difference is in how you retrieve them.

In my sample, I used a BASH script to set the output variable based on a parameter passed into the pipeline, but you can create output variables using scripts or tasks

  - stage: SetupStage
    displayName: 'Setup Stage'
    jobs:
      - job: SetupJob
        displayName: 'Setup Job'
        steps:
          - checkout: none
          - bash:  |
              set -e # need to avoid trailing " being added to the variable https://github.com/microsoft/azure-pipelines-tasks/issues/10331
              echo "##vso[task.setvariable variable=MyVar;isOutput=true]${{parameters.value}}"
            name: SetupStep
            displayName: 'Setup Step'

Possible ways to access a stage dependency variable

There are two basic ways to access stage dependency variables, both using array objects

stageDependencies.STAGENAME.JOBNAME.outputs['STEPNAME.VARNAME']
dependencies.STAGENAME.outputs['JOBNAME.STEPNAME.VARNAME']

Which one you use, in which place, and whether via a local alias is the complexity

How to access a stage dependency in a script?

To access a stage dependency variable in a script, or a task, there are two key requirements

  • The stage containing the consuming job and hence script/task, must be set as dependant on the stage that created the output variable
  • You have to declare a local alias for the value in the stageDependencies array within the consuming stage. This local alias will be used as the local name by scripts and tasks

Once this is configured you can access the variable like any other local YAML variable

  - stage: Show_With_Dependancy
    displayName: ‘Show Stage With dependancy’
    dependsOn:
      - SetupStage
    variables:
      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs[‘SetupStep.MyVar’]]
    jobs:
      - job: Job
        displayName: ‘Show Job With dependancy’
        steps:
        - bash: |
              echo “localMyVarViaStageDependancies - $(localMyVarViaStageDependancies)”

Tip: If you are having a problem with the value not being set for a stage dependency variable look in the pipeline execution log, at the job level, and check the ‘Job preparation parameters’ section to see what is being evaluated. This will show if you are using the wrong array object, or have a typo, as any incorrect declarations evaluate as null

How to use a stage dependency as a stage condition

You can use stage dependency variables as controlling conditions for running a stage. In this use-case you use the dependencies array and not the stagedependencies used when aliasing variables.

  - stage: Show_With_Dependancy_Condition
    condition: and (succeeded(), eq (dependencies.SetupStage.outputs['SetupJob.SetupStep.MyVar'], 'True'))
    displayName: 'Show Stage With dependancy Condition'

From my experiments for this use-case, you don’t seem to need the DependsOn entry to decare the stage that exposed the output variable for this to work. So, this is very useful for complex pipelines where you want to skip a later stage based on a much earlier stage for which there is no direct dependency.

A side effect of using a stage condition is that many subsequent stages have to have their execution conditions edited as you cannot rely on the default completion stage state succeeded. This is because the prior stages could now be succeeded or skipped. Hence all following stages need to use the condition

condition: and( not(failed()), not(canceled()))

How to use a stage dependency as a job condition

To avoid the need to alter all the subsequent stage’s execution conditions you can set a condition at the job or task level. Unlike setting the condition at that stage level, you have to create a local alias (see above) and check the condition on that

  - stage: Show_With_Dependancy_Condition_Job
    displayName: 'Show Stage With dependancy Condition'
    dependsOn:
      - SetupStage
    variables:
      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs['SetupStep.MyVar']]
    jobs:
      - job: Job
        condition: and (succeeded(),
          eq (variables.localMyVarViaStageDependancies, 'True'))
        displayName: 'Show Job With dependancy'

This technique will work for both Agent-based and Agent-Less (Server) jobs

A warning though, if your job makes use of an environment with a manual approval, the environment approval check is evaluated before the job condition. This is probably not what you are after, so if using conditions with environments that use manual approvals then the condition is probably best set at the stage level, with the knock-on issues of states of subsequent stages as mentioned above.

An alternative, if you are just using the environment for manual approval, is to look at using an AgentLess job with a manual approval. AgentLess job manual approvals are evaluated after the job condition, so do not suffer the same problem.

If you need to use a stage dependency variable in a later stage, as a job condition or script variable, but do not wish to add a direct dependency between the stages, you could consider ‘republishing’ the variable as an output of the intermedia stage(s)

  - stage: Intermediate_Stage
    dependsOn:
      - SetUpStage
    variables:
      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs['SetupStep.MyVar']]
    jobs:
      - job: RepublishMyVar
       steps:
          - checkout: none
          - bash:  |
              set -e # need to avoid trailing " being added to the variable https://github.com/microsoft/azure-pipelines-tasks/issues/10331
              echo "##vso[task.setvariable variable=MyVar;isOutput=true]$( localMyVarViaStageDependancies)"
            name: RepublishStep

Summing Up

So I hope this post will help you, and the future me, navigate the complexities of stage variables

You can find the YAML for the test harness I have been using in this GitHub GIST

Setting Azure DevOps ‘All Repositories’ Policies via the CLI

The Azure DevOps CLI provides plenty of commands to update Team Projects, but it does not cover all things you might want to set. A good example is setting branch policies. For a given repo you can set the policies using the Azure Repo command eg:

az repos policy approver-count update --project <projectname> --blocking true --enabled true --branch main --repository-id <guid> --minimum-approver-count w --reset-on-source-push true  --creator-vote-counts false --allow-downvotes false    

However, you hit a problem if you wish to set the ‘All Repositories’ policies for a Team Project. The issue is that the above command requires a specific –project parameter.

I can find no way around this using any published CLI tools, but using the REST API there is an option.

You could of course check the API documentation to work out the exact call and payload. However, I usually find it quicker to perform the action I require in the Azure DevOps UI and monitor the network traffic in the browser developer tools to see what calls are made to the API.

Using this technique, I have created the following script that sets the All Repositories branch policies.

Note that you can use this same script to set a specific repo’s branch policies by setting the repositoryId in the JSON payloads.