Using Azure DevOps Stage Dependency Variables with Conditional Stage and Job Execution

I have been doing some work with Azure DevOps multi-stage YAML pipelines using stage dependency variables and conditions. They can get confusing quickly, you need one syntax in one place and another elsewhere.

So, here are a few things I have learnt...

What are stage dependency variables?

Stage Dependencies are the way you define which stage follows another in a multi-stage YAML pipeline. This is as opposed to just relying on the order they appear in the YAML file, the default order. Hence, they are critical to creating complex pipelines.

Stage Dependency variables are the way you can pass variables from one stage to another. Special handling is required, as you can’t just use the ordinary output variables (which are in effect environment variables on the agent) as you might within a job as there is no guarantee the stages and jobs are running on the same agent.

For stage dependency variables, is not how you create output variables, that does not differ from the standard manner, the difference is in how you retrieve them.

In my sample, I used a BASH script to set the output variable based on a parameter passed into the pipeline, but you can create output variables using scripts or tasks

 1 - stage: SetupStage
 2    displayName: 'Setup Stage'
 3    jobs:
 4      - job: SetupJob
 5        displayName: 'Setup Job'
 6        steps:
 7          - checkout: none
 8          - bash:  |
 9              set -e # need to avoid trailing " being added to the variable https://github.com/microsoft/azure-pipelines-tasks/issues/10331
10              echo "##vso[task.setvariable variable=MyVar;isOutput=true]${{parameters.value}}"
11            name: SetupStep
12            displayName: 'Setup Step' 

Possible ways to access a stage dependency variable

There are two basic ways to access stage dependency variables, both using array objects

1stageDependencies.STAGENAME.JOBNAME.outputs['STEPNAME.VARNAME']
2dependencies.STAGENAME.outputs['JOBNAME.STEPNAME.VARNAME']

Which one you use, in which place, and whether via a local alias is the complexity

How to access a stage dependency in a script?

To access a stage dependency variable in a script, or a task, there are two key requirements

• The stage containing the consuming job and hence script/task, must be set as dependant on the stage that created the output variable
• You have to declare a local alias for the value in the stageDependencies array within the consuming stage. This local alias will be used as the local name by scripts and tasks

Once this is configured you can access the variable like any other local YAML variable

 1 - stage: Show_With_Dependancy
 2    displayName: ‘Show Stage With dependancy’
 3    dependsOn:
 4      - SetupStage
 5    variables:
 6      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs[‘SetupStep.MyVar’]]
 7    jobs:
 8      - job: Job
 9        displayName: ‘Show Job With dependancy’
10        steps:
11        - bash: |
12              echo “localMyVarViaStageDependancies - $(localMyVarViaStageDependancies)” 

Tip: If you are having a problem with the value not being set for a stage dependency variable look in the pipeline execution log, at the job level, and check the ‘Job preparation parameters’ section to see what is being evaluated. This will show if you are using the wrong array object, or have a typo, as any incorrect declarations evaluate as null

How to use a stage dependency as a stage condition

You can use stage dependency variables as controlling conditions for running a stage. In this use-case you use the dependencies array and not the stagedependencies used when aliasing variables.

1 - stage: Show_With_Dependancy_Condition
2    condition: and (succeeded(), eq (dependencies.SetupStage.outputs['SetupJob.SetupStep.MyVar'], 'True'))
3    displayName: 'Show Stage With dependancy Condition' 

From my experiments for this use-case, you don’t seem to need the DependsOn entry to decare the stage that exposed the output variable for this to work. So, this is very useful for complex pipelines where you want to skip a later stage based on a much earlier stage for which there is no direct dependency.

A side effect of using a stage condition is that many subsequent stages have to have their execution conditions edited as you cannot rely on the default completion stage state succeeded. This is because the prior stages could now be succeeded or skipped. Hence all following stages need to use the condition

1condition: and( not(failed()), not(canceled()))

How to use a stage dependency as a job condition

To avoid the need to alter all the subsequent stage's execution conditions you can set a condition at the job or task level. Unlike setting the condition at that stage level, you have to create a local alias (see above) and check the condition on that

 1 - stage: Show_With_Dependancy_Condition_Job
 2    displayName: 'Show Stage With dependancy Condition'
 3    dependsOn:
 4      - SetupStage
 5    variables:
 6      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs['SetupStep.MyVar']]
 7    jobs:
 8      - job: Job
 9        condition: and (succeeded(),
10          eq (variables.localMyVarViaStageDependancies, 'True'))
11        displayName: 'Show Job With dependancy' 

This technique will work for both Agent-based and Agent-Less (Server) jobs

A warning though, if your job makes use of an environment with a manual approval, the environment approval check is evaluated before the job condition. This is probably not what you are after, so if using conditions with environments that use manual approvals then the condition is probably best set at the stage level, with the knock-on issues of states of subsequent stages as mentioned above.

An alternative, if you are just using the environment for manual approval, is to look at using an AgentLess job with a manual approval. AgentLess job manual approvals are evaluated after the job condition, so do not suffer the same problem.

If you need to use a stage dependency variable in a later stage, as a job condition or script variable, but do not wish to add a direct dependency between the stages, you could consider ‘republishing’ the variable as an output of the intermedia stage(s)

 1 - stage: Intermediate_Stage
 2    dependsOn:
 3      - SetUpStage
 4    variables:
 5      localMyVarViaStageDependancies : $[stageDependencies.SetupStage.SetupJob.outputs['SetupStep.MyVar']]
 6    jobs:
 7      - job: RepublishMyVar
 8       steps:
 9          - checkout: none
10          - bash:  |
11              set -e # need to avoid trailing " being added to the variable https://github.com/microsoft/azure-pipelines-tasks/issues/10331
12              echo "##vso[task.setvariable variable=MyVar;isOutput=true]$( localMyVarViaStageDependancies)"
13            name: RepublishStep 

Summing Up

So I hope this post will help you, and the future me, navigate the complexities of stage variables

You can find the YAML for the test harness I have been using in this GitHub GIST