New release of my ‘Generate Parameters.xml tools to add support for app.config files

I recently released an updated version of my Generate Parameters.XML tool for Visual Studio. This release adds support for generating parameters.xml files from app.config files as well as web.config files

Why you might ask why add support for app.config files when the parameters.xml model is only part of WebDeploy?

Well, at Black Marble we like the model of updating a single file using a tokenised set of parameters from within our DevOps CI/CD pipelines. It makes it easy to take release variables and write them, at deploy time, into a parameters.xml file to be injected into a machine’s configuration. We wanted to extend this to configuring services and the like where for example a DLL based service is configured with a mycode.dll.config file

The injection process of the parameters.xml into a web.config file is automatically done as part of the WebDeploy process (or a VSTS extension wrapping WebDeploy), but if you want to use a similar model for app.config files then you need some PowerShell.

For example, if we have the app.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <applicationSettings>
     <Service.Properties.Settings>
       <setting name="Directory1" serializeAs="String">
         <value>C:ABC1111</value>
       </setting>
       <setting name="Directory2" serializeAs="String">
         <value>C:abc2222</value>
       </setting>
     </Service.Properties.Settings>
   </applicationSettings>
   <appSettings>
     <add key="AppSetting1" value="123" />
     <add key="AppSetting2" value="456" />
   </appSettings>
     <startup> 
         <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6" />
     </startup>
</configuration>

My extension generates a tokenised parameters.xml file

<parameters>
  <parameter name="AppSetting1" description="Description for AppSetting1" defaultvalue="__APPSETTING1__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/appSettings/add[@key='AppSetting1']/@value" />
  </parameter>
  <parameter name="AppSetting2" description="Description for AppSetting2" defaultvalue="__APPSETTING2__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/appSettings/add[@key='AppSetting2']/@value" />
  </parameter>
  <parameter name="Directory1" description="Description for Directory1" defaultvalue="__DIRECTORY1__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory1']/value/text()" />
  </parameter>
  <parameter name="Directory2" description="Description for Directory2" defaultvalue="__DIRECTORY2__" tags="">
     <parameterentry kind="XmlFile" scope="\App.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory2']/value/text()" />
  </parameter>
 </parameters>

The values in this parameters.xml file can be updated using a CI/CD replace tokens task, we use Colin’s ALM Corner Build & Release Tools, Replace Tokens, in exactly the same way as we would for a web.config

Finally the following PowerShell can be used to update the app.config from this parameters.xml

Thus giving a consistent way of updating configuration files for both web.config and app.config files

Smarter Cities Summit

I am delighted to be joining SixPivot, our amazing Australian partner, in taking part in the Safer Cities Summit in Brisbane this week.  We are launching tuServ, our award-winning policing solution at the event, into the Australian market, as the first step to taken it global!

I’ll be joined by Faith Rees, SixPivot’s CEO, and I am so excited to share with the Australian public sector the transformation that tuServ makes possible inside an organisation. 

The Public Sector Network’s Safer Cities Summit is an Australian based event focused on bringing together key civic safety focussed government stakeholders from around the world to tackle these important challenges. It is tailored towards Australian and international public safety, emergency and disaster response and urban resilience related stakeholders.


b.

Versioning your ARM templates within a VSTS CI/CD pipeline with Semantic Versioning

I wrote a post recently Versioning your ARM templates within a VSTS CI/CD pipeline. I realised since writing it that it does not address the issue of if you wish to version your ARM Templates using Semantic Versioning. My JSON versioning task I used did not support the option of not extracting a numeric version number e.g. 1.2.3.4 from a VSTS build number. To address this limitation I have modified my Version JSON file task to address.

This change to my task allows it to be used with the GitVersion VSTS task to manage the semantic versioning. For more details on GitVersion see the project documentation.

Hence, I my now able to generate a version number using GitVersion and pass this in to the versioning task directly using a build variable.

  • Add the GitVersion task at the start of the build, with its default parameters
  • Add my JSON versioning task with default parameters apart from
    • Version Number set to $(GitVersion.SemVer)
    • Use Version Number without Processing (Advanced) checked
    • Filename Pattern (Advanced) set to azuredeploy.json
    • Field to update (Advanced) set to contentVersion

image

In the logs you see output similar to the following

Source Directory: E:Build2_work361s
Filename Pattern: azuredeploy.json
Version Number/Build Number: 0.1.0-unstable.843
Use Build Number Directly: true
Version Filter to extract build number: d+.d+.d+.d+
Version Format for JSON File: {1}.{2}.{3}
Field to update (all if empty): contentVersion
Output: Version Number Parameter Name: OutputedVersion
Using the provided build number without any further processing
JSON Version Name will be: 0.1.0-unstable.843
Will apply 0.1.0-unstable.843 to 12 files.
Updating the field 'contentVersion' version
Existing Tag: contentVersion": "1.0.0.0"
Replacement Tag: contentVersion": "0.1.0-unstable.843"
…

What I wish I had known when I started developing Lability DevTest Lab Environments

At Black Marble we have been migrating our DevTest labs to from on-premises TFS Lab Management to a mixture of on-premise and Azure hosted Lability defined Labs as discussed by Rik Hepworth on his blog. I have only been tangentially involved in this effort until recently, consuming the labs but not creating the definitions.

So this post is one of those I do where I don’t want to forget things I learnt the hard way, or to put it another way asking Rik or Chris after watching a 2 hour environment deploy fail for the Xth time.

  • You can’t log tool much. The log files are your friends, both the DSC ones and any generated by tools triggered by DSC. This is because most of the configuration process is done during boots so there is no UI to watch.
  • The DSC log is initially created in working folder the .MOF file is in on the target VM; but after a reboot (e.g. after joining a domain) the next and subsequent DSC log files are created in  C:WindowsSystem32ConfigurationConfigurationStatus
  • Make sure you specify the full path for any bespoke logging you do, relative paths make it too easy to lose the log file
  • Stupid typos get you every time, many will be spotted when the MOF file is generated, but too many such as ones in command lines or arguments are only spotted when you deploy an environment. Also too many of these don’t actually cause error messages, they just mean nothing happens. So if you expect a script/tool to be run and it doesn’t check the log and the definition for mismatches in names.
  • If you are using the Package DSC Resource to install an EXE or MSI couple of gotcha’s
    • For MSIs the ProductName parameter must exactly match the one in the MSI definition, and this must match the GUID ProductCode.  Both of these can be found using the Orca tool

      image

    • Package MongoDb {

      PsDscRunAsCredential = $DomainCredentialsAtDomain

      DependsOn = '[Package]VCRedist'

      Ensure = 'Present'

      Arguments = "/qn /l*v c:bootstrapMongoDBInstall.log INSTALLLOCATION=`"C:Program FilesMongoDBServer3.6`""

      Name = "MongoDB 3.6.2 2008R2Plus SSL (64 bit)"

      Path = "c:bootstrapmongodb-win32-x86_64-2008plus-ssl-3.6.2-signed.msi"

      ProductId = "88B5F0D8-0692-4D86-8FF4-FB3CDBC6B40F"

      ReturnCode = 0

      }

    • For EXEs the ProductName does not appear to be as critical, but you still need the Product ID. You can get this with PowerShell on a machine that already has the EXE installed
    • Get-WmiObject Win32_Product | Format-Table IdentifyingNumber, Name, Version
  • I had network issues, they could mostly be put does to incorrect Network Address Translation. In my case this should have been setup when Lability was initially configured, the commands ran OK creating a virtual switch and NetNat, but I ended up with a Windows failback network address of 169.x.x.x when I should have had an address of 192.168.x.x on my virtual switch. So if in doubt check the settings on your virtual switch, in the Windows ‘Networking and Share Center’ before you start doubting your environment definitions.

Hope these pointers help others, as well as myself, next time Lability definitions are written

DDD (South) is back again

 DDD12_thumb

After last year’s successful DDD, it will be in Reading for its thirteenth outing, on Saturday, 23th of June.

Once again DDD is free for all to attend, thanks to some great sponsors – if you would like to be sponsor DDD, please email ddd@blackmarble.com.

As ever I would love to see new speakers and new topics in the DDD line up, so please submit a session, this year for new speakers we will be trying to get support before the event to ensure they are super successful.

Session submission will open soon watch @DeveloperDay for more news

I look forward to seeing you all in June and we will be announcing more DDD dates in the next few weeks

b.

Versioning your ARM templates within a VSTS CI/CD pipeline

Updated 3 Feb 2018Also see Versioning your ARM templates within a VSTS CI/CD pipeline with Semantic Versioning

Azure Resource Templates (ARM) allow your DevOps infrastructure deployments to be treated as ‘content as code’. So infrastructure definitions can be stored in source control.

As with any code it is really useful to know which version you have out in production. Now a CI/CD process and its usage logs can help here, but just having a version string stored somewhere accessible on the production systems is always useful.

In an ARM Template this can be achieved using the ‘content version’ field in the template (see documentation for more detail on this file). The question becomes how best to update this field with a version number?

The solution I used was a VSTS JSON Versioning Task I had already created to update the template’s .JSON definition file. I popped this task at the start of my ARM templates CI build process and it set the value prior to the storage of the template as a build artifact used within the CD pipeline

image

Creating test data for my Generate Release Notes Extension for use in CI/CD process

As part of the continued improvement to my CI/CD process I needed to provide a means so that whenever I test my Generate Release Notes Task, within it’s CI/CD process, new commits and work item associations are made. This is required because the task only picks up new commits and work items since the last successful running of a given build. So if the last release of the task extension was successful then the next set of tests have no associations to go in the release notes, not exactly exercising all the code paths!

In the past I added this test data by hand, a new manual commit to the repo prior to a release; but why have a dog and bark yourself? Better to automate the process.

This can done using a PowerShell file, run inline or stored in the builds source repo and run within a VSTS build. The code is shown below, you can pass in the required parameters, but I set sensible default for my purposes

For this PowerShell code to work you do need make some security changes to allow the build agent service user to write to the Git repo. This is documented by Microsoft.

The PowerShell task to run this code is placed in a build as the only task

image

This build is then triggered as part of the release process

image

Note that the triggering of this build has to be such that it runs on a non-blocking build agent as discussed in my previous posts. In my case I trigger the build to add the extra commits and work items just before triggering the validation build on my private Azure hosted agent.

Now, there is no reason you can’t just run the PowerShell directly within the release if you wanted to. I chose to use a build so that the build could be reused between different VSTS extension CI/CD pipelines; remember I have two Generate Release Note Extensions, PowerShell and NodeJS Based.

So another step to fully automating the whole release process.

How I fixed my problem that my VSTS Build Extension was too big to upload to the Marketplace

Whist adding a couple of new tasks to my VSTS Manifest Versioning Extension I hit the problem that VSIX package became too big to upload to the Marketplace.

The error I saw in my CI/CD VSTS pipeline was

##vso[task.logissue type=error;]error: 
Failed Request: Bad Request(400) - 
The extension package size '23255292 bytes' exceeds the 
maximum package size '20971520 bytes'

This extension now contains  eleven tasks, four of which are now NodeJS based as opposed to PowerShell. The issue here is whereas PowerShell tasks are usually a file or two of code and maybe a PSM module; NodeJS based ones, as well as my logic, always have a Node_Modules folder full of NPM modules needed for production use. This fact had caused a good deal of bloat in the VSIX package.

The solution was to address my poor management of NPM modules. As many of the versioning tasks are similar in logical structure i.e.

  1. They get a list of files
  2. Extract a version number from the build number
  3. Then apply this to one or more files in a product/task specific manner

there has been some cut and paste coding. This means that I have NPM modules in the tasks package.json file that were not needed for a given task. I could manually address this but there is an NPM module to help, DepCheck.

First install the DepCheck module

npm install depcheck –g

then run depcheck from the command line whist within your task’s folder. This returns a list of modules listed in the package.json that are not referenced in the code files. These can then be removed from the package.json.  e.g. I saw

Unused dependencies
* @types/node
* @types/q
* Buffer
* fs
* request
* tsd
Unused devDependencies
* @types/chai
* @types/mocha
* @types/node
* mocha-junit-reporter
* ts-loader
* ts-node
* typings

The important ones to focus on are the first block (non-development references), as these are the ones that are packaged with the production code in the VSIX; I was already pruning the node_module folder of development dependencies prior to creating the VSIX to remove devDependancies using the command

npm prune –production

I did find some of the listed modules strange, as I knew they really were needed and a quick test of removing them did show the code failed if they were missing. These are what depchecks documentation calls false alerts.

I found I could remove the @type/xxx and tsd references, which were the big ones, that are only needed in development when working in TypeScript. Once these were removed for all four of my NodeJS based tasks my VSIX dropped in size from 22Mb to 7Mb. So problem solved.

Added a new JSON version task to my VSTS Version Extension

In response to requests on the VSTS Marketplace I have added a pair of tasks to added/edit entries in a .JSON format files.

The first is for adding a version to a file like a package.json file e.g.

{
"name": "myapp",
"version": "1.0.0",
"license": "MIT"
}

The second is designed for angular environment.ts file e.g.

export const environment = {
production: true,
version: '1.0.0.0'
};

But I bet people find other uses, they always do.

You can find the extension in the marketplace, you need 1.31.x or later to see the new versioner tasks

Announcing a new VSTS Extension for Starting and Stopping Azure DevTest Labs VMs

Background

I have recently been posting on using Azure to host private VSTS build/release agents to avoid agent queue deadlocking issues with more complex release pipelines.

One of the areas discussed is reducing cost of running a private agent in Azure by only running the private agent within a limited time range, when you guess it might be needed. I have done this using DevTest Labs Auto Start and Auto Stop features. This works, but is it not better to only start the agent VM when it is actually really needed, not when you guess it might be? I need this private agent only when working on my VSTS extensions, not something I do everyday. Why waste CPU cycles that are never used?

New VSTS Extension

I had expected there would already be a VSTS  extension to Start and Stop DevTest Lab VMs, but the Microsoft provided extension for DevTest Labs only provides tasks for the creation and deletion of VMs within a lab.

So I am pleased to announce the release of my new DevTest Labs VSTS Extension to fill this gap, adding tasks to start and stop a DevTest Lab VM on demand from within a build or a release.

My Usage

I have been able to use the tasks in this extension to start my private Azure hosted agent only when I need it for functional tests within a release.

However, they could equally be used for a variety of different testing scenarios where any form of pre-built/configured VMs needs to be started or stopped as opposed to slower processes of creating/deploying a new deployment of a DevTest lab VM.

In may case I added an extra agent phases to my release pipeline to start the VM prior to it being needed.

image

I could also have used another agent phase to stop the VM once the tests were completed. However, I made the call to leave the VM running and let DevTest Labs’ Auto Stop shut it down at the end of the day. The reason for this is that VM start up and shutdown is still fairly slow, a minute or two, and I often find I need to run a set of function tests a few times during my development cycle; so it is a bit more efficient to leave the VM running until the end of the day. Only taking the start-up cost once.

You may have course have different needs, hence providing both the Start and Stop Tasks

Development

This new extension aims to act as a supplement to the Microsoft provided Azure DevTest Lab Extension. Hence to make development and adoption easier, it uses exactly the same source code structure and task parameters as the Microsoft provided extension. The task parameters being:

  • Azure RM Subscription – Azure Resource Manager subscription to configure before running.
  • Source Lab VM ID – Resource ID of the source lab VM. The source lab VM must be in the selected lab, as the custom image will be created using its VHD file. You can use any variable such as $(labVMId), the output of calling Create Azure DevTest Labs VM, that contains a value in the form /subscriptions/{subId}/resourceGroups/{rgName}/providers/Microsoft.DevTestLab/labs/{labName}/virtualMachines/{vmName}.

The issue I had was that the DevTest Labs PowerShell API did not provide a command to start or stop a VM in a lab. I needed to load the Azure PowerShell library to use the Invoke-AzureRmResourceAction  command. This requires you first call Login-AzureRmAccount to authenticate prior to calling the actual Invoke-AzureRmResourceAction required. This required a bit of extra code to get and reuse the AzureRM endpoint to find the authentication details.

# Get the parameters
$ConnectedServiceName = Get-VstsInput -Name "ConnectedServiceName"
# Get the end point from the name passed as a parameter
$Endpoint = Get-VstsEndpoint -Name $ConnectedServiceName -Require
# Get the authentication details
$clientID = $Endpoint.Auth.parameters.serviceprincipalid
$key = $Endpoint.Auth.parameters.serviceprincipalkey
$tenantId = $Endpoint.Auth.parameters.tenantid
$SecurePassword = $key | ConvertTo-SecureString -AsPlainText -Force
$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $clientID, $SecurePassword
# Authenticate
Login-AzureRmAccount -Credential $cred -TenantId $tenantId -ServicePrincipal

Important to note that for this code to work you have to set the task’s task.json to run PowerShell3 and package the Powershell VSTS API module in with the task.

"execution": {
  "PowerShell3": {
     "target": "$(currentDirectory)\StartVM.ps1",
     "argumentFormat": "",
     "workingDirectory": "$(currentDirectory)"
    }
  }

If the folder structure is correct changing to PowerShell3 will automatically load the required module from the tasks ps_module folder

In Summary

I have certainly found this extension useful, and I have leant more that I had expect I would about VSTS endpoints and Azure authentication.

Hope it is useful to you too.