Azure DevOps Services & Server Alerts DSL – an alternative to TFS Aggregator?

Whilst listening to a recent  Radio TFS it was mentioned that TFS Aggregator uses the C# SOAP based Azure DevOps APIs; hence needed a major re-write as these APIs are being deprecated.

Did you know that there was a REST API alternative to TFS Aggregator?

My Azure DevOps Services & Server Alerts DSL is out there, and has been for a while, but I don’t think used by many people. It aims to do the same as TFS Aggregator, but is based around Python scripting.

However, I do have to say it is more limited in flexibility as it has only been developed for my (and a few of my clients needs), but its an alternative that is based on the REST APIs. 

Scripts are of the following form, this one changes the state of a work item if all it children are done

import sys
# Expect 2 args the event type and a value unique ID for the wi
if sys.argv[0] == "workitem.updated" : 
    wi = GetWorkItem(int(sys.argv[1]))
    parentwi = GetParentWorkItem(wi)
    if parentwi == None:
        LogInfoMessage("Work item '" + str(wi.id) + "' has no parent")
    else:
        LogInfoMessage("Work item '" + str(wi.id) + "' has parent '" + str(parentwi.id) + "'")

        results = [c for c in GetChildWorkItems(parentwi) if c["fields"]["System.State"] != "Done"]
        if  len(results) == 0 :
            LogInfoMessage("All child work items are 'Done'")
            parentwi["fields"]["System.State"] = "Done"
            UpdateWorkItem(parentwi)
            msg = "Work item '" + str(parentwi.id) + "' has been set as 'Done' as all its child work items are done"
            SendEmail("richard@blackmarble.co.uk","Work item '" + str(parentwi.id) + "' has been updated", msg)
            LogInfoMessage(msg)
        else:
            LogInfoMessage("Not all child work items are 'Done'")
else:
	LogErrorMessage("Was not expecting to get here")
	LogErrorMessage(sys.argv)

I have recently done a fairly major update to the project. The key changes are:

  • Rename of project, repo, and namespaces to reflect Azure DevOps (the namespace change is a breaking change for existing users)
  • The scripts that are run can now be
    • A fixed file name for the web instance running the service
    • Based on the event type sent to the service
    • Use the subscription ID, thus allowing many scripts (new)
  • A single instance of the web site running the events processor can now handle calls from many Azure DevOps instances.
  • Improved installation process on Azure (well at least tried to make the documentation clearer and sort out a couple of MSDeploy issues)

Full details are on the project can be seen on the solutions WIKI, maybe you will find it of use. Let me know if the documentation is good enough

Major new release of my VSTS Cross Platform Extension to build Release Notes

Today I have released a major new release, V2, of my VSTS Cross Platform Extension to build release notes. This new version is all down to the efforts of Greg Pakes who has completely re-written the task to use newer VSTS APIs.

A minor issue is that this re-write has introduced a couple of breaking changes, as detailed below and on the project wiki

  • oAuth script access has to be enabled on the agent running the task

image

  • There are minor changes in the template format, but for the good, as it means both TFVC and GIT based releases now use a common template format. Samples can be found in the project repo

Because of the breaking changes, we made the decision to release both V1 and V2 of the task in the same extension package, so not forcing anyone to update unless they wish to. A technique I have not tried before, but seems to work well in testing.

Hope people still find the task of use and thanks again to Greg for all the work on the extension

Using VSTS Gates to help improve my deployment pipeline of VSTS Extensions to the Visual Studio Marketplace

My existing VSTS CI/CD process has a problem that the deployment of a VSTS extension, from the moment it is uploaded to when it’s tasks are available to a build agent, is not instantiation. The process can potentially take a few minutes to roll out. The problem this delay causes is a perfect candidate for using VSTS Release Gates; using the gate to make sure the expected version of a task is available to an agent before running the next stage of the CD pipeline e.g waiting after deploying a private build of an extension before trying to run functional tests.

The problem is how to achieve this with the current VSTS gate options?

What did not work

My first thought was to use the Invoke HTTP REST API gate, calling the VSTS API https://<your vsts instance name>.visualstudio.com/_apis/distributedtask/tasks/<GUID of Task>. This API call returns a block of JSON containing details about the deployed task visible to the specified VSTS instance. In theory you can parse this data with a JSONPATH query in the gates success criteria parameter to make sure the correct version of the task is deployed e.g. eq($.value[?(@.name == “BuildRetensionTask”)].contributionVersion, “1.2.3”)

However, there is a problem. At this time the Invoke HTTP REST API gate task does not support the == equality operator in it’s success criteria field. I understand this will be addressed in the future, but the fact it is currently missing is a block to my current needs.

Next I thought I could write a custom VSTS gate. These are basically ‘run on server’ tasks with a suitably crafted JSON manifest. The problem here is that this type of task does not allow any code (Node.JS or PowerShell) to be run. They only have a limited capability to invoke HTTP APIs or write messages to service bus. So I could not implement the code I needed to process the API response. So another dead end.

What did work

The answer, after a suggestion from the VSTS Release Management team at Microsoft, was to try the Azure Function gate.

To do this I created a new Azure Function. I did this using the Azure Portal, picking the consumption billing model, C# and securing the function with a function key, basically the default options.

I then added the C# function code (stored in GitHub), to my newly created Azure Function. This function code takes

  • The name of the VSTS instance
  • A personal access token (PAT) to access the VSTS instance
  • The GUID of the task to check for
  • And the version to check for

It then returns a JSON block with true or false based on whether the required task version can be found. If any of the parameters are invalid an API error is returned

By passing in this set of arguments my idea was that a single Azure Function could be used to check for the deployment of all my tasks.

Note: Now I do realise I could also create a release pipeline for the Azure Function, but I chose to just create it via the Azure Portal. I know this is not best practice, but this was just a proof of concept. As usual the danger here is that this proof of concept might be one of those that is too useful and lives forever!

To use the Azure Function

Using the Azure function is simple

    • Added an Azure Function gate to a VSTS release
    • Set the URL parameter for the Azure Function. This value can be found from the Azure Portal. Note that you don’t need the Function Code query parameter in the URL as this is provided with the next gate parameter. I chose to use a variable group variable for this parameter so it was easy to reuse between many CD pipelines
    • Set the Function Key parameter for the Azure Function, again you get this from the Azure Portal. This time I used a secure variable group variable
    • Set the Method parameter to POST
    • Set the Header content type as JSON
{
      "Content-Type": "application/json"
}
    • Set the Body to contain the details of the VSTS instance and Task to check. This time I used a mixture of variable group variables, release specific variables (the GUID) and environment build/release variables. The key here is I got the version from the primary release artifact $(BUILD.BUILDNUMBER) so the correct version of the tasks is tested for automatically
{
     "instance": "$(instance)",
     "pat": "$(pat)",
     "taskguid": "$(taskGuid)",
     "version": "$(BUILD.BUILDNUMBER)"
}
  • Finally set  the Advanced/Completion Event to ApiResponse with the success criteria of
    eq(root['Deployed'], 'true')

Once this was done I was able to use the Azure function as a VSTS gate as required

image

Summary

So I now have a gate that makes sure that for a given VSTS instance a task of a given version has been deployed.

If you need this functionality all you need to do is create your own Azure Function instance, drop in my code and configure the VSTS gate appropriately.

When equality == operator becomes available for JSONPATH in the REST API Gate I might consider a swap back to a basic REST call, it is less complex to setup, but we shall see. The Azure function model does appear to work well

New release of my ‘Generate Parameters.xml tools to add support for app.config files

I recently released an updated version of my Generate Parameters.XML tool for Visual Studio. This release adds support for generating parameters.xml files from app.config files as well as web.config files

Why you might ask why add support for app.config files when the parameters.xml model is only part of WebDeploy?

Well, at Black Marble we like the model of updating a single file using a tokenised set of parameters from within our DevOps CI/CD pipelines. It makes it easy to take release variables and write them, at deploy time, into a parameters.xml file to be injected into a machine’s configuration. We wanted to extend this to configuring services and the like where for example a DLL based service is configured with a mycode.dll.config file

The injection process of the parameters.xml into a web.config file is automatically done as part of the WebDeploy process (or a VSTS extension wrapping WebDeploy), but if you want to use a similar model for app.config files then you need some PowerShell.

For example, if we have the app.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <applicationSettings>
     <Service.Properties.Settings>
       <setting name="Directory1" serializeAs="String">
         <value>C:ABC1111</value>
       </setting>
       <setting name="Directory2" serializeAs="String">
         <value>C:abc2222</value>
       </setting>
     </Service.Properties.Settings>
   </applicationSettings>
   <appSettings>
     <add key="AppSetting1" value="123" />
     <add key="AppSetting2" value="456" />
   </appSettings>
     <startup> 
         <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6" />
     </startup>
</configuration>

My extension generates a tokenised parameters.xml file

<parameters>
  <parameter name="AppSetting1" description="Description for AppSetting1" defaultvalue="__APPSETTING1__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/appSettings/add[@key='AppSetting1']/@value" />
  </parameter>
  <parameter name="AppSetting2" description="Description for AppSetting2" defaultvalue="__APPSETTING2__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/appSettings/add[@key='AppSetting2']/@value" />
  </parameter>
  <parameter name="Directory1" description="Description for Directory1" defaultvalue="__DIRECTORY1__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory1']/value/text()" />
  </parameter>
  <parameter name="Directory2" description="Description for Directory2" defaultvalue="__DIRECTORY2__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory2']/value/text()" />
  </parameter>
 </parameters>

The values in this parameters.xml file can be updated using a CI/CD replace tokens task, we use Colin’s ALM Corner Build & Release Tools, Replace Tokens, in exactly the same way as we would for a web.config

Finally the following PowerShell can be used to update the app.config from this parameters.xml

Thus giving a consistent way of updating configuration files for both web.config and app.config files

Creating test data for my Generate Release Notes Extension for use in CI/CD process

As part of the continued improvement to my CI/CD process I needed to provide a means so that whenever I test my Generate Release Notes Task, within it’s CI/CD process, new commits and work item associations are made. This is required because the task only picks up new commits and work items since the last successful running of a given build. So if the last release of the task extension was successful then the next set of tests have no associations to go in the release notes, not exactly exercising all the code paths!

In the past I added this test data by hand, a new manual commit to the repo prior to a release; but why have a dog and bark yourself? Better to automate the process.

This can done using a PowerShell file, run inline or stored in the builds source repo and run within a VSTS build. The code is shown below, you can pass in the required parameters, but I set sensible default for my purposes

For this PowerShell code to work you do need make some security changes to allow the build agent service user to write to the Git repo. This is documented by Microsoft.

The PowerShell task to run this code is placed in a build as the only task

image

This build is then triggered as part of the release process

image

Note that the triggering of this build has to be such that it runs on a non-blocking build agent as discussed in my previous posts. In my case I trigger the build to add the extra commits and work items just before triggering the validation build on my private Azure hosted agent.

Now, there is no reason you can’t just run the PowerShell directly within the release if you wanted to. I chose to use a build so that the build could be reused between different VSTS extension CI/CD pipelines; remember I have two Generate Release Note Extensions, PowerShell and NodeJS Based.

So another step to fully automating the whole release process.

Announcing a new VSTS Extension for Starting and Stopping Azure DevTest Labs VMs

Background

I have recently been posting on using Azure to host private VSTS build/release agents to avoid agent queue deadlocking issues with more complex release pipelines.

One of the areas discussed is reducing cost of running a private agent in Azure by only running the private agent within a limited time range, when you guess it might be needed. I have done this using DevTest Labs Auto Start and Auto Stop features. This works, but is it not better to only start the agent VM when it is actually really needed, not when you guess it might be? I need this private agent only when working on my VSTS extensions, not something I do everyday. Why waste CPU cycles that are never used?

New VSTS Extension

I had expected there would already be a VSTS  extension to Start and Stop DevTest Lab VMs, but the Microsoft provided extension for DevTest Labs only provides tasks for the creation and deletion of VMs within a lab.

So I am pleased to announce the release of my new DevTest Labs VSTS Extension to fill this gap, adding tasks to start and stop a DevTest Lab VM on demand from within a build or a release.

My Usage

I have been able to use the tasks in this extension to start my private Azure hosted agent only when I need it for functional tests within a release.

However, they could equally be used for a variety of different testing scenarios where any form of pre-built/configured VMs needs to be started or stopped as opposed to slower processes of creating/deploying a new deployment of a DevTest lab VM.

In may case I added an extra agent phases to my release pipeline to start the VM prior to it being needed.

image

I could also have used another agent phase to stop the VM once the tests were completed. However, I made the call to leave the VM running and let DevTest Labs’ Auto Stop shut it down at the end of the day. The reason for this is that VM start up and shutdown is still fairly slow, a minute or two, and I often find I need to run a set of function tests a few times during my development cycle; so it is a bit more efficient to leave the VM running until the end of the day. Only taking the start-up cost once.

You may have course have different needs, hence providing both the Start and Stop Tasks

Development

This new extension aims to act as a supplement to the Microsoft provided Azure DevTest Lab Extension. Hence to make development and adoption easier, it uses exactly the same source code structure and task parameters as the Microsoft provided extension. The task parameters being:

  • Azure RM Subscription – Azure Resource Manager subscription to configure before running.
  • Source Lab VM ID – Resource ID of the source lab VM. The source lab VM must be in the selected lab, as the custom image will be created using its VHD file. You can use any variable such as $(labVMId), the output of calling Create Azure DevTest Labs VM, that contains a value in the form /subscriptions/{subId}/resourceGroups/{rgName}/providers/Microsoft.DevTestLab/labs/{labName}/virtualMachines/{vmName}.

The issue I had was that the DevTest Labs PowerShell API did not provide a command to start or stop a VM in a lab. I needed to load the Azure PowerShell library to use the Invoke-AzureRmResourceAction  command. This requires you first call Login-AzureRmAccount to authenticate prior to calling the actual Invoke-AzureRmResourceAction required. This required a bit of extra code to get and reuse the AzureRM endpoint to find the authentication details.

# Get the parameters
$ConnectedServiceName = Get-VstsInput -Name "ConnectedServiceName"
# Get the end point from the name passed as a parameter
$Endpoint = Get-VstsEndpoint -Name $ConnectedServiceName -Require
# Get the authentication details
$clientID = $Endpoint.Auth.parameters.serviceprincipalid
$key = $Endpoint.Auth.parameters.serviceprincipalkey
$tenantId = $Endpoint.Auth.parameters.tenantid
$SecurePassword = $key | ConvertTo-SecureString -AsPlainText -Force
$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $clientID, $SecurePassword
# Authenticate
Login-AzureRmAccount -Credential $cred -TenantId $tenantId -ServicePrincipal

Important to note that for this code to work you have to set the task’s task.json to run PowerShell3 and package the Powershell VSTS API module in with the task.

"execution": {
  "PowerShell3": {
     "target": "$(currentDirectory)\StartVM.ps1",
     "argumentFormat": "",
     "workingDirectory": "$(currentDirectory)"
    }
  }

If the folder structure is correct changing to PowerShell3 will automatically load the required module from the tasks ps_module folder

In Summary

I have certainly found this extension useful, and I have leant more that I had expect I would about VSTS endpoints and Azure authentication.

Hope it is useful to you too.

Creating a VSTS build agent on an Azure DevLabs Windows Server VM with no GUI

Updates


As I posted recently I have been trying to add more functional tests to the VSTS based release CI/CD pipeline for my VSTS Extensions, and as I noted depending on how you want to run your tests e.g. trigger sub-builds, you can end up with scheduling deadlocks where a single build agent is scheduling the release and trying to run a new build. The answer is to use a second build agent in a different agent pool e.g. if the release is running on the Hosted build agent use a private build agent for the sub-build, or of course just pay for more hosted build instances.

The problem with a private build agent is where to run it. As my extensions are a personal project I don’t have a corporate Hyper-V server to run any extra private agents on, as I would have for an company projects. My MVP MSDN Azure benefits are the obvious answer, but I want any agents to be cheap to run, so I don’t burn through all my MSDN credits for a single build agent.

To this end I created a Windows Server 2016 VM in DevLabs (I prefer to create my VMs in DevLabs as it makes it easier tidying up of my Azure account) using an A0 sizing VM. This is tiny so cheap; I don’t intend to ever do a build on this agent, just schedule releases, so need to install few if any tools, so the size should not be an issue. To further reduce costs I used the auto start and stop features on the VM so it is only running during the hours I might be working. So I get an admittedly slow and limited private build agent but for less that $10 a month.

As the VM is small it makes sense to not run a GUI. This means when you RDP to the new VM you just get a command prompt. So how do you get the agent onto the VM and setup? You can’t just open a browser to VSTS or cut and paste a file via RDP, and I wanted to avoid the complexity of having to open up PowerShell remoting on the VM.

The process I used was as follows:

  1. In VSTS I created a new Agent Pool for my Azure hosted build agents
  2. In the Azure portal, DevLabs I created a new Windows Server 2016 (1709) VM
  3. I then RDP’d to my new Azure VM, in the open Command Prompt I ran PowerShell
    powershell
  4. As I was in my users home directory, I  cd’d into the downloads folder
    cd downloads
  5. I then ran the following PowerShell command to download the agent (you can get the current URI for the agent from your VSTS Agent Pool ‘Download Agent’ feature, but an old version will do as it will auto update.
    invoke-webrequest -UseBasicParsing -uri https://github.com/Microsoft/vsts-agent/releases/download/v2.124.0/vsts-agent-win7-x64-2.124.0.zip -OutFile vsts-agent-win7-x64-2.124.0.zip
  6. You can then follow the standard agent setup instructions from the VSTS Agent Pool ‘Download Agent’ feature
    mkdir agent ; cd agent
    PS
    Add-Type -AssemblyName System.IO.Compression.FileSystem ; [System.IO.Compression.ZipFile]::ExtractToDirectory(“$HOMEDownloadsvsts-agent-win7-x64-2.124.0.zip”, “$PWD”)
  7. I then configured the agent to run as a service, I exited back to the command prompt to do this this, so the commands were
    exit
    config.cmd

I now had an other build agent pool to use in my CI/CD pipelines at a reasonable cost, and the performance was not too bad either.


	

Major update to my CI/CD process for VSTS extensions

As time passes I have found there is a need for more cross platform VSTS extensions as there is more uptake of VSTS beyond it’s historic Microsoft platform based roots.

Historically most of my extensions have been Powershell based. Now this is not a fundamental problem for cross platform usage. this is due to the availability of  Powershell Core. However, Typescript and Node.JS is a better fit I think in many cases. This has caused me to revise the way I structure my repo and build my VSTS extensions to provide a consistent understandable process. My old Gulp based process  for Typescript was too complex and inconsistent between tasks, it even confused me!  My process revisions have been documented in my vNextBuild Github’s WIKI, so I don’t propose too repeat the bulk of the content here.

That said, it is worth touching on why I did not use YO VSTS. If I were starting this project again I would certainly look at this tool to build out my basic repo structure. Also I think I would look at a separate repo per VSTS extension, as opposed to putting them all in one repo. However, this project pre-dates the availability of  YO VSTS , hence the structure I have is different. Also as people have forked the repo I don’t intend to introduce breaking changes by splitting the repo. That all said my Typescript/Node.JS process is heavily influenced by the structures and NPM script used in YO VSTS, so people should find the core processes familiar i.e.

  1. At the command prompt
  2. CD into an extension folder you wish to build (this contains the package.json file)
  3. Get all the NPM packages npm install as defined in the package.json
  4. Run TSLink and transpile the TypeScript to Node.JS. There is a single command npm run build to do this, but they can be rerun individually using npm run lint and npm run transpile
  5. Run an unit tests you have npm run test-no-logging (Note the npm run test script is used by the CI/CD process to dump the test results for uploading to VSTS logging. I tend to find when working locally that just dumping the test results to the console is enough.
  6. Package the the resultant .JS files npm run package. This script does two jobs, it remove any NPM modules that are set as -savedev i.e. only used in development and not production and also copies the required files to the correct locations to be packaged into a VSIX file. NOTE after this command is run you need to rerun npm install to reinstall the development NPM packages prior to be able to run tests etc. locally

This is complement by an enhance VSTS CI/CD process that has a far greater focus on automated testing as well as packaging the VSIX files and releasing to the VSTS marketplace

image

  Check the vNextBuild Github’s WIKI for more details

Putting a release process around my VSTS extension development

Updated: 5th Aug 2016 added notes in PublisherID


 

I have been developing a few VSTS/TFS build related extensions and have published a few in the VSTS marketplace. This has all been a somewhat manual process, a mixture of Gulp and PowerShell has helped a bit, but I decided it was time to try to do a more formal approach. To do this I have used Jesse Houwing’s VSTS Extension Tasks.

Even with this set of tasks I am not sure what I have is ‘best practice’, but it does work. The doubt is due to the way the marketplace handles revisions and preview flags. What I have works for me, but ‘your mileage may differ’

My Workflow

The core of my workflow is that I am building the VSIX package twice, once as a private package and the other as a public one. They both contain the same code and have the same version number, they differ in only visibility flags

I am not using a the preview flag options at all. I have found they do not really help me. My workflow is to build the private package, upload it and test it by sharing it with a test VSTS instance. if all is good publish the matched public package on the marketplace. In this model there is no need to use a preview, it just adds complexity I don’t need.

This may not be true for everyone.

Build

The build’s job is to take the code, set the version number and package it into multiple VSIX package.

  1. First I have the vNext build get my source from my GitHub repo.
  2. I add two build variables $(Major) and $(Minor) that I use to manually manage my version number
  3. I set my build number format to $(Major).$(Minor).$(rev:r), so the final .number is incremented until I choose to increment the major or minor version.
  4. I then use one of Jesse’s tasks to package the extension multiple times using the extension tag model parameter. Each different package step uses different Visibility settings (circled in red). I also set the version, using the override options, to the $(Build.BuildNumber) (circled in green)image
  5. [Updated Aug 2016] Set the PublisherID and ExtensionID on the tasks, using a pair of build variables is a good idea here to avoid entering strings twice. It is important thay the PublisherID is entered with the correct case – it is case sensitive within the marketplace. Strange things happend of the PublisherID in a VSIX package differ from the one registered on the marketplace
  6. As I am using the VSTS hosted build agent I also need to make sure I check the install Tfx-cli in the global setting section
  7. I then add a second identical publish task, but this time there is no tag set and the visibility is set to public.
  8. Finally I use a ‘publish build artifacts’ task to copy the VSIX packages to a drop location

Release

So now I have multiple VSIX packages I can use the same family of tasks to create a release pipeline.

I create a new release linked to be a Continuous Deployment of the previously created build and set its release name format to Release-$(Build.BuildNumber)

My first environment uses three tasks, all using the option – to work from a VSIX package.

Note In all cases I am using the VSIX path in the format $(System.DefaultWorkingDirectory)/GenerateReleaseNotes.Master/vsix/<package name>-<tag>-$(Build.BuildNumber).vsix. I am including the build number variable in the path as I chose to put all the packages in a single folder, so path wildcards are not an option as the task would not know which package to use unless I alter my build to put one VSIX package per folder.

My tasks for the first environment are

  1. Publish VSTS Extension – using my private package so it is added as a private package to the marketplace
  2. Share VSTS Extension – to my test VSTS account
  3. Install VSTS Extension – to my test VSTS account

For details in the usage of these tasks and setting up the link to the VSTS Marketplace see Jesse’s wiki

If I only intend a extension to ever be private this is enough. However I want to make mine public so I add a second environment that has manual pre-approval (so I have to confirm the public release)

This environment only needs single task

  1. Publish VSTS Extension – using my public package so it is added as a public package to the marketplace

I can of course add other tasks to this environment maybe send a Tweet or email to publicise the new version’s release

Summary

So now I have a formal way to release my extensions. The dual packaging model means I can publish two different versions at the same time one privately and the other public

image

It is now just a case of moving all my extensions over to the new model.

Though I am still interested to hear what other people view are? Does this seem a reasonable process flow?