The random thoughts of Richard Fennell on technology and software development
Today Microsoft made a big announcement, VSTS is now Azure DevOps.
The big change is they have split VSTS into 5 services you can use together or independently, including Azure Pipelines for CI/CD – free for open source and available in the GitHub CI marketplace.
An important thing to note is that IT IS NOT JUST FOR AZURE.
Don’t be afraid of the name. There a wide range of connectors to other cloud providers such as AWS and Google Cloud, as will as many other DevOps tools
The documentation for the VSTS WIKI suggests you can embed a video in a VSTS WIKI using the markdown/HTML
<video src="_media/vstswiki_mid.mp4" width=400 controls> </video>
Problem is that this does not seem to work, the MP4 just does not appear, you get an empty video player.
However, if you swap to a full URL it does work e.g.
<video src="https://sec.ch9.ms/ch9/7247/7c8ddc1a-348b-4ba9-ab61-51fded6e7247/vstswiki_high.mp4" width=400 controls> </video>
This is a problem if you wish to store media locally in your WIKI
The workaround is to either place the MP4 file in some URL accessible location e.g. some Azure web space (not really addressing the problem), or more usefully use the VSTS API to get the file out the repo that backs the WIKI.
The format of the HTML tag becomes
<video src="https://vstsinstance.visualstudio.com/MyTeamProject/_apis/git/repositories/MyTeamProject.wiki/Items?path=_media%2Fvstswiki_high.mp4 width=400" controls> </video>
This will get the current version of the file on default branch, you can add extra parameters to to specify versions and branches if required as per the API documentation.
So not a perfect solution as you have to think about branches and versions, they are not handled automatically, but at least it does work
When you want to register a build agent with VSTS, you use the VSTS instance’s URL and a user’s Personal Access Token (PAT). Whilst doing this today I connected to the VSTS instance OK but got the error “Agent pool not found”.when I was asked to pick the agent pool to add the new agent to.
As the user who’s PAT I was using was a Build Administrator I was a bit confused, but then I remembered to check their user access level. It was set to Stakeholder, once this was changed to Basic I was able to register the agent without use.
Also, so as to not use up a Basic license, when I did not need to, I swapped the user back to being a Stakeholder once the agent was registered. This can be done as the token used for the actual build is not the one used to register it, but one assigned at build time by VSTS.
You have always been able to customise your Team Projects in TFS, by editing a host of XML files, but it was not a pleasant experience. In VSTS a far more pleasant to use web based inherited customisation model was added, much to, I think, most administrators relief.
If you used the TFS DB migration service you ended up with a VSTS instance full of the the XML style team projects, and you were stuck there, with no way to change these to the new inherited mode, that is until now as Microsoft have released to preview a conversion tool.
I have been trying this tool to migrate all our active XML based team projects to Inherited equivalents, and it has worked very well for me, but I have to say, we don’t have any hugely complex customisations, so your mileage may vary depending on how much you modified your XML based team project templates.
However, I wanted to go further than the basic process as documented
Since moving to VSTS all our new team project have been created using an inherited template based on Scrum. I wanted to move all our active XML based team project to this standardised template. This took a little extra work.
The basic process is easy as to change Inherited Process you
REMEMBER: A really nice touch (with XML and Inherited templates) is that you can just switch back if you don’t like the result, no data is lost when you swap templates, but some of it might be hidden as fields are not shown on the new work item types.
The problem I had was that our old XML based team project templates had different customisation to our current inherited standard. To address this I needed to
So once done, I was then able to migrate all the team project to my target process template.
So a very nice experience, and one that now means we can make sure all our team projects use the same set of customisation. No longer do we need to worry about customising in XML and Inherited models.
Whilst writing some training material for VSTS I hit a problem creating a TFVC workspace. I was using VS2017, linking a TFVC Repo to a local folder. I was connecting to the VSTS instance using an MSA.
In Team Explorer, when I came to do a ‘Map & Get’ to map the source locations I got a ‘TF14061: The workspace ws_1_18;Project Collection Build Service does not exist’ error
Strange error, which I could see no obvious reason for. Turns out the work around was just to press the ‘Advanced’ link/button and accept the defaults
I have a VSTS build extension that can apply a VSTS generated build number to Android APK packages. This takes a VSTS build number and generates, and applies, the Version Name (a string) and Version Code (an integer) to the APK file manifest.
The default parameters mean that the behaviour of this task is to assume (using a regular expression) the VSTS build number has at least three fields major.minor.patch e.g. 1.2.3, and uses the 1.2 as the Version Name and the 3 as the Version Code.
Now, it is important to note that the Version Code must be a integer between 1 and 2100000000 and for the Google Play Store it must be incrementing between versions.
So maybe these default parameter values for this task are not the best options?
When we use the Android Manifest Versioning task for our tuServ Android packages we use different parameter values, but we recently found these values still cause a problem.
Our VSTS build generates build numbers with four parts $(Major).$(Minor).$(Year:yy)$(DayOfYear).$(rev:r)
So we end up with build numbers in the form 1.2.18101.1
The Android version task is set in the build to make
The problem is if we do more than 9 builds in a day, which is likely due to our continuous integration process, and release one of the later builds to the Google Play store, then the next day any build with a lower revision than 9 cannot be released to the store as its Version Code is lower than the previously published one e.g.
So the second Version Code is 10x smaller, hence the package cannot be published.
The answer in the end was straightforward and found by one of our engineers Peter (@sarkimedes). It was to change the final block of the VSTS build number to $(rev:rrr), as detailed in the VSTS documentation. Thus zero padding the revision from .1 to .001. This allows up to 1000 builds per day before the problem of altering the Version Code order of magnitude problem occurs. Obviously, if you think you might do more than 1000 internal builds in a day you could zero pack as many digits as you want.
So using the new build version number
So a nice fix without any need to alter the Android Manifest Versioning task’s code. However, changing the default Version Code parameter to {1}{2}{3} is probably advisable.
Today I have released a major new release, V2, of my VSTS Cross Platform Extension to build release notes. This new version is all down to the efforts of Greg Pakes who has completely re-written the task to use newer VSTS APIs.
A minor issue is that this re-write has introduced a couple of breaking changes, as detailed below and on the project wiki
Because of the breaking changes, we made the decision to release both V1 and V2 of the task in the same extension package, so not forcing anyone to update unless they wish to. A technique I have not tried before, but seems to work well in testing.
Hope people still find the task of use and thanks again to Greg for all the work on the extension
Azure is a highly resilient service, and VSTS has excellent SLAs. However, a question that is often asked is ‘How do I backup my VSTS instance?’.
The simple answer is you don’t. Microsoft handle keeping the instance up, patched and serviceable. Hence, there is no built in means for you to get a local copy of all your source code, work items or CI/CD definitions. Though there have been requests for such a service.
This can be an issue for some organisations, particularly for source control, where there can be a need to have a way to keep a private copy of source code for escrow, DR or similar purposes.
To address this issue I decided to write a PowerShell script to download all the GIT and TFVC source in a VSTS instance. The following tactics were used
So that the process was run on regular basis I designed it to be run within a VSTS build. Again here I had choices:
I chose the former, so a single scheduled build could backup all my VSTS instances by running the script a number of time with different parameters
To use this script you just pass in
The obvious next step is to convert the PowerShell script to be a VSTS Extension, at this point it would make sense to make it optional to use a provided PAT or the System Access Token.
Also I could add code to allow a number of multiple cycling backups to be taken e.g. keep the last 3 backups
These are maybe something for the future, but they don’t really seems a good return in investment at this time to package up a working script as an extensions just for a single VSTS instance.
I recently, whilst changing a CI/CD release pipeline, updated what was previously a private version of a VSTS extension in the VSTS Marketplace with a version of the VSIX package set to be public.
Note, in my CI/CD process I have a private and public version of each extension (set of tasks), the former is used for functional testing within the CD process, the latter is the one everyone can see.
So, this meant I had two public versions of the same extension, confusing.
Turns out you can’t change a public extension back to be private, either via the UI or by uploading a corrected VSIX. Also you can’t delete any public extension that has ever been downloaded, and my previously private one had been downloaded once, by me for testing.
So my only option was to un-publish the previously private extension so only the correct version was visible in the public marketplace.
This meant I had to also alter my CI/CD process to change the extensionID of my private extension so I could publish a new private version of the extension.
Luckily, as all the GUIDs for the tasks within the extension did not change once I had installed the new version of the extension I had mispublished in my test VSTS instance my pipeline still worked.
Only downside is I am left with an un-publish ‘dead’ version listed in my private view of the marketplace. This is not a problem, just does not look ‘neat and tidy’
My existing VSTS CI/CD process has a problem that the deployment of a VSTS extension, from the moment it is uploaded to when it’s tasks are available to a build agent, is not instantiation. The process can potentially take a few minutes to roll out. The problem this delay causes is a perfect candidate for using VSTS Release Gates; using the gate to make sure the expected version of a task is available to an agent before running the next stage of the CD pipeline e.g waiting after deploying a private build of an extension before trying to run functional tests.
The problem is how to achieve this with the current VSTS gate options?
My first thought was to use the Invoke HTTP REST API gate, calling the VSTS API https://<your vsts instance name>.visualstudio.com/_apis/distributedtask/tasks/<GUID of Task>. This API call returns a block of JSON containing details about the deployed task visible to the specified VSTS instance. In theory you can parse this data with a JSONPATH query in the gates success criteria parameter to make sure the correct version of the task is deployed e.g. eq($.value[?(@.name == “BuildRetensionTask”)].contributionVersion, “1.2.3”)
However, there is a problem. At this time the Invoke HTTP REST API gate task does not support the == equality operator in it’s success criteria field. I understand this will be addressed in the future, but the fact it is currently missing is a block to my current needs.
Next I thought I could write a custom VSTS gate. These are basically ‘run on server’ tasks with a suitably crafted JSON manifest. The problem here is that this type of task does not allow any code (Node.JS or PowerShell) to be run. They only have a limited capability to invoke HTTP APIs or write messages to service bus. So I could not implement the code I needed to process the API response. So another dead end.
The answer, after a suggestion from the VSTS Release Management team at Microsoft, was to try the Azure Function gate.
To do this I created a new Azure Function. I did this using the Azure Portal, picking the consumption billing model, C# and securing the function with a function key, basically the default options.
I then added the C# function code (stored in GitHub), to my newly created Azure Function. This function code takes
It then returns a JSON block with true or false based on whether the required task version can be found. If any of the parameters are invalid an API error is returned
By passing in this set of arguments my idea was that a single Azure Function could be used to check for the deployment of all my tasks.
Note: Now I do realise I could also create a release pipeline for the Azure Function, but I chose to just create it via the Azure Portal. I know this is not best practice, but this was just a proof of concept. As usual the danger here is that this proof of concept might be one of those that is too useful and lives forever!
Using the Azure function is simple
{
"Content-Type": "application/json"
}
{
"instance": "$(instance)",
"pat": "$(pat)",
"taskguid": "$(taskGuid)",
"version": "$(BUILD.BUILDNUMBER)"
}
eq(root['Deployed'], 'true')
Once this was done I was able to use the Azure function as a VSTS gate as required
So I now have a gate that makes sure that for a given VSTS instance a task of a given version has been deployed.
If you need this functionality all you need to do is create your own Azure Function instance, drop in my code and configure the VSTS gate appropriately.
When equality == operator becomes available for JSONPATH in the REST API Gate I might consider a swap back to a basic REST call, it is less complex to setup, but we shall see. The Azure function model does appear to work well