BM-Bloggers

The blogs of Black Marble staff

New version of my VSTS Generate Release Notes extension - now supports Builds and Release

I am pleased to announce that I have just made public on the VSTS marketplace a new version of my VSTS Generate Release Notes extension.

This new version now supports both VSTS/TFS vNext Builds and vNext Releases. The previous versions only supported the generation of release notes as part of a build.

The adding of support for release has meant I have had to rethink the internals of how the templates is process as well as the way templates are passed into the task and where results are stored

  • You can now provide a template as a file (usually from source control) as before, but also as an inline property. The latter is really designed for Releases where there is usually no access to source control, only to build artifact drops (though you could put the template in one of these if you wanted)
  • With a build the obvious place to put the release notes file is in the drops location. For a release there is no such artifact drop location, so I just leave the releases notes on the release agent, it is up to the user to get this file copied to a sensible location for their release process.

To find out more check out the documentation on my GitHub repo and have a look at my sample templates to get you started generating release notes

WSUS Non-Functional After KB3159706 Installed

Consider the following scenario:

  • You have WSUS installed on either Windows Server 2012 or 2012 R2
  • You install KB3159706

In this situation, WSUS fails to start correctly and thus fails to function.

There are additional steps that are required to configure this update once it is installed. The steps can be found in KB3159706.

Note: If using database mirroring or the SUSDB is part of an AlwaysOn Availability Group, this must be undone before performing the actions described in KB3159706 as a schema update is required for the database.

Putting a release process around my VSTS extension development

Updated: 5th Aug 2016 added notes in PublisherID


 

I have been developing a few VSTS/TFS build related extensions and have published a few in the VSTS marketplace. This has all been a somewhat manual process, a mixture of Gulp and PowerShell has helped a bit, but I decided it was time to try to do a more formal approach. To do this I have used Jesse Houwing’s VSTS Extension Tasks.

Even with this set of tasks I am not sure what I have is ‘best practice’, but it does work. The doubt is due to the way the marketplace handles revisions and preview flags. What I have works for me, but ‘your mileage may differ’

My Workflow

The core of my workflow is that I am building the VSIX package twice, once as a private package and the other as a public one. They both contain the same code and have the same version number, they differ in only visibility flags

I am not using a the preview flag options at all. I have found they do not really help me. My workflow is to build the private package, upload it and test it by sharing it with a test VSTS instance. if all is good publish the matched public package on the marketplace. In this model there is no need to use a preview, it just adds complexity I don’t need.

This may not be true for everyone.

Build

The build’s job is to take the code, set the version number and package it into multiple VSIX package.

  1. First I have the vNext build get my source from my GitHub repo.
  2. I add two build variables $(Major) and $(Minor) that I use to manually manage my version number
  3. I set my build number format to $(Major).$(Minor).$(rev:r), so the final .number is incremented until I choose to increment the major or minor version.
  4. I then use one of Jesse’s tasks to package the extension multiple times using the extension tag model parameter. Each different package step uses different Visibility settings (circled in red). I also set the version, using the override options, to the $(Build.BuildNumber) (circled in green)

    image
  5. [Updated Aug 2016] Set the PublisherID and ExtensionID on the tasks, using a pair of build variables is a good idea here to avoid entering strings twice. It is important thay the PublisherID is entered with the correct case - it is case sensitive within the marketplace. Strange things happend of the PublisherID in a VSIX package differ from the one registered on the marketplace
  6. As I am using the VSTS hosted build agent I also need to make sure I check the install Tfx-cli in the global setting section
  7. I then add a second identical publish task, but this time there is no tag set and the visibility is set to public.
  8. Finally I use a ‘publish build artifacts’ task to copy the VSIX packages to a drop location

Release

So now I have multiple VSIX packages I can use the same family of tasks to create a release pipeline.

I create a new release linked to be a Continuous Deployment of the previously created build and set its release name format to Release-$(Build.BuildNumber)

My first environment uses three tasks, all using the option - to work from a VSIX package.

Note In all cases I am using the VSIX path in the format $(System.DefaultWorkingDirectory)/GenerateReleaseNotes.Master/vsix/<package name>-<tag>-$(Build.BuildNumber).vsix. I am including the build number variable in the path as I chose to put all the packages in a single folder, so path wildcards are not an option as the task would not know which package to use unless I alter my build to put one VSIX package per folder.

My tasks for the first environment are

  1. Publish VSTS Extension – using my private package so it is added as a private package to the marketplace
  2. Share VSTS Extension – to my test VSTS account
  3. Install VSTS Extension – to my test VSTS account

For details in the usage of these tasks and setting up the link to the VSTS Marketplace see Jesse’s wiki

If I only intend a extension to ever be private this is enough. However I want to make mine public so I add a second environment that has manual pre-approval (so I have to confirm the public release)

This environment only needs single task

  1. Publish VSTS Extension – using my public package so it is added as a public package to the marketplace

I can of course add other tasks to this environment maybe send a Tweet or email to publicise the new version’s release

Summary

So now I have a formal way to release my extensions. The dual packaging model means I can publish two different versions at the same time one privately and the other public

image

It is now just a case of moving all my extensions over to the new model.

Though I am still interested to hear what other people view are? Does this seem a reasonable process flow?

Upgrading BlogEngine to 3.3

I have just completed the upgrade of our Blog Server to BlogEngine 3.3. This upgrade is a bit more complex than the usual upgrade as between 3.2 to 3.3 there is a change to Razor views for all the widgets. This means you need to remove all the old widgets you have and re-add them using the new razor equivalents.

As our blog is backed by SQL, this mean a SQL script to clear down the old widgets, then a manual add of the new versions on each blog we have on our server. One point to note, if using SQL you do need to get BlogEngine 3.3 from its GitHub repo (at the time of writing, I am sure this will change) as after the formal 3.3 release on CodePlex there is a fix for an issue that stopped the editing of widget properties.

So first experiences with 3.3?

Seems much more responsive, so all is looking good

Azure Web Apps-Deploying Java Servlets to Azure App Service Web Apps

If you are considering to move to hosting your websites in Azure but either have a lot of legacy applications written in Java or your organisation is Java focussed, then Azure App Services provide the option to host Java code (Java Servlets, JSPs etc.) in the same way that they can host .NET code (ASP.NET Web Api, Forms, MVC etc.)

To test this, I have taken a pre-built WAR file containing a single Java Servlet, and see how much effort was required to host it in an Azure Web App.

The approach to hosting Java is as follows:

1. Create the Web App.

2. Go into the Web App, Enable the Java runtime and select your application server (Tomcat and Jetty are available).

image image

3. Upload your WAR file to the Web App. I chose FTP, but there are a number of options for publishing.  To reiterate, the process of publishing a Java Web App is exactly the same as if you were publishing a .NET Web App (except that you don’t have the option of using Visual Studio to publish).  Note: put your WAR file in the “site\wwroot\webapps” folder.  This isn’t immediately obvious and can be one of two places depending on how the web app was provisioned.  See this article for more information.

image

4. Confirm it as running.

That’s all there is to it.

Granted, this is a simple scenario, but Azure web apps have the capability to reach on to your on-premise network using things like Site-to-site VPN, ExpressRoute or Hybrid Connections to give you access to resources like databases, line-of-business systems etc. on your network.

Updates to my StyleCop task for VSTS/TFS 2015.2

Tracking the current version of StyleCop is a bit awkward. Last week I got an automated email from CodePlex saying 4.7.52.0 had been released . I thought this was the most up to date version, so upgraded my StyleCop command line wrapper and my VSTS StyleCop task from 4.7.47.0 to 4.7.52.0.

However, I was wrong about the current version. I had not realised that the StyleCop team  had forked the code onto GitHub. GitHub is now the home of the Visual Studio 2015 and C# 6 development of StyleCop, while Codeplex remains the home of the legacy Visual Studio versions. I had only upgraded to a legacy patch version, not the current version.

So I upgraded my StyleCop Command Line tool and my VSTS StyleCop task to wrapper 4.7.59.0, thus I think bringing me up to date.

Using Coded UI from Windows 8.1 to Windows 10 UWP

Recently one of our bigger projects which has a Windows 8.1 client has just been migrated to Windows 10 UWP. The application is quite big and is rapidly changing every week. When it comes to testing we need to make sure that all of the old functionality is still working as expected. To do this we run a set of nightly builds which execute Coded UI tests.When converting the application to Windows 10 we was not sure if the Coded UI tests would just work or if we needed to do some work….so we just ran them!

At first all was looking well as the application launched however nothing else happened to our disappointment. So, I decided to debug one of the tests to find out what was happening.

The way that we was launching the application was my just grabbing the application tile by its ID (guid) as you can see from the screenshot below:

image

This worked fine and then when we was trying to find one of the controls it thought about it for a very long time and then couldn’t find it. When I looked further into the child controls of the main application window it did seem to be picking some up, however none of the controls had an automation id that had initialised. So I decided to try searching for the control by name which also did not work. I was very confused as I could see a long list of controls as child items from the main application.

 

At this point I needed some more feedback from the UI so I decided to basically loop through each child control and highlight it so I could see what controls it was actually picking up and see where they was located on the UI.

I was very disappointed to see that none of the buttons highlighted, but what seemed to be random pieces of the UI.

 

We often come across some controls on the application that the Coded UI finds difficult to interact with. To help the search methods we can filter the search by providing some more attributes. Search configuration and search properties filters through the list of controls as you can see below:

image

So instead of trying this on the actual control itself (i.e button or combo box) I decided to add these filters when grabbing the main application window. Whilst debugging I could actually see the controls where found and the automation id’s where now initialised. This looked more promising!

 

I decided to let the debugger continue and see if the test got any further…which it did! It now seemed to be behaving as I would expect it to. However I did run into another problem. In our application we use quite a lot of dialogs. The way that these dialogs are created in the Windows 8.1 application was by using Direct UI controls. In Windows 10 UWP the dialogs are built up using Xaml controls which make interacting with them a lot easier. Because the application uses a lot of dialogs I thought it would be easier to make a helper method that i can call from different tests:

image

Again using the filters to make sure that the test can find the dialog. The first action of getting the child controls gets the parent areas of the dialog, such as title, or buttons. The second action will then get the child controls from the parent container and I know that each time the tests interact with a dialog the buttons will be in this parent control of the dialog.

 

This actually made my life a little easier as I knew that all dialogs would be using the same controls and automation id’s. The last challenge that we faced was that now most of our Windows 10 builds have now been migrated to vNext. This means that we will now have to set up a release pipeline to get the tests running again…..another blog post to follow!

Azure Logic Apps-The template language function 'json' parameter is not valid.

This is a follow up from my original blog post Azure Logic Apps–Parsing JSON message from service bus

If you see the following error from a Logic App using Service Bus trigger:

{"code":"InvalidTemplate","message":"Unable to process template language expressions in action 'Next_action' inputs at line '1' and column '11': 'The template language function 'json' parameter is not valid. The provided value 'eydlbWFpbCcgOiAndGVzdEB0ZXN0LmNvbSd9' cannot be parsed: 'Unexpected character encountered while parsing value: e. Path '', line 0, position 0.'. Please see https://aka.ms/logicexpressions#json for usage details.'."}

It’s because messages that from service bus topic triggers are now presented as base 64 encoded messages rather than plain text.  Fortunately the fix is straightforward, Logic Apps give you a decodeBase64 function that you can use in the Logic App. 

You need to switch the Logic App to to code view, and use the decodeBase64 function before parsing the json.  So

image

becomes (note the call to decodeBase64 before the call to json):

image

Build 2016 - Overview

Overall I must say that I found this to be one of the best Build's yet. The keynotes had people cheering and applauding almost none stop, the content in the sessions was exemplary and the expo hall and was fun to spend time in with lots of interesting stands to talk to people on.

My only regret is that I didn't have as much time to spend in a actual sessions due to my running around doing other things. Though I have been watching sessions that I missed at https://channel9.msdn.com/Events/Build/2016                                             to catch up and continue learning.

If you want to see more photos of our time at Build please check out our Facebook page at

https://www.facebook.com/Calicowind/?fref=ts

Or if you want to follow us on twitter you can find us at

https://twitter.com/Calicowind                                                                   Follow us to keep track of all our fun exploits in programming

Thank you :D

 

 

 

 

 

How to build a connection string from other parameters within MSDeploy packages to avoid repeating yourself in Release Management variables

Whilst working with the new Release Management features in VSTS/TFS 2015.2 I found I needed to pass in configuration variables i.e. server name, Db name, UID and Password to create a SQL server via an Azure Resource Management Template release step and a connection string to the same SQL instance for a web site’s web.config, set using an MSDeploy release step using token replacement (as discussed in this post)

Now I could just create RM configuration variables for both the connection string and ARM settings,

image

 

However, this seems wrong for a couple of reason

  1. You should not repeat your self, too easy to get the two values out of step
  2. I don’t really want to obfuscate the whole of a connection string in RM, when only a password really needs to be hidden (note the connection string variable is not set as secure in the above screenshot)

What did not work

I first considered nesting the RM variables, e.g. setting a the connection string variable to be equal to ‘Server=tcp: $(DatabaseServer).database.windows.net,1433;Database=$(DatabaseName)….’, but this does not give the desired results, the S(DatabaseServer) and $(DatabaseName) variables are not expanded at runtime, you just get a string with the variable names in it.

How I got want I was after….

(In this post as a sample I am using the Fabrikam Fiber solution. This means I need to provide a value for the FabrikamFiber-Express connection string)

I wanted to build the connection string from the other variables in the MSDeploy package. So to get the behaviour I want…

  1. In Visual Studio load the Fabrikam web site solution.
  2. In the web project, use the publish option to create a publish profile use the ‘WebDeploy package’ option.
  3. If you publish this package you end up with a setparameter.xml file containing the default connection string
    <setParameter name="FabrikamFiber-Express-Web.config Connection String" value="Your value”/>
    Where ‘your value’ is the value you set in the Publish wizard. So to use this I would need to pass in a whole connection string, where I only want to pass parts of this string
  4. To add bespoke parameters to an MSDeploy package you add a parameter.xml file to the project in Visual Studio (I wrote a Visual Studio Extension that help add this file, but you can create it by hand). My tool will create the parameters.xml file based on the AppSettings block of the projects Web.config. So if you have a web.config containing the following
    <appSettings>
        <add key="Location" value="DEVPC" />
      </appSettings>
    It will create a parameters.xml file as follows
    <?xml version="1.0" encoding="utf-8"?>
    <parameters>
      <parameter defaultValue="__LOCATION__" description="Description for Location" name="Location" tags="">
        <parameterentry kind="XmlFile" match="/configuration/appSettings/add[@key='Location']/@value" scope="\\web.config$" />
      </parameter>
    </parameters>
  5. If we publish at this point we will get a setparameters.xml file containing
    <?xml version="1.0" encoding="utf-8"?>
    <parameters>
      <setParameter name="IIS Web Application Name" value="__Sitename__" />
      <setParameter name="Location" value="__LOCATION__" />
      <setParameter name="FabrikamFiber-Express-Web.config Connection String" value="__FabrikamFiberWebContext__" />
    </parameters>
    This is assuming I used the publish wizard to set the site name to __SiteName__ and the DB connection string to __FabrikamFiberWebContext__
  6. Next step is to add my DB related parameters to the paramaters.xml file, this I do by hand, my tool does not help
    <?xml version="1.0" encoding="utf-8"?>
    <parameters>
      <parameter defaultValue="__LOCATION__" description="Description for Location" name="Location" tags="">
        <parameterentry kind="XmlFile" match="/configuration/appSettings/add[@key='Location']/@value" scope="\\web.config$" />
      </parameter>

      <parameter name="Database Server" defaultValue="__sqlservername__"></parameter>
      <parameter name="Database Name" defaultValue="__databasename__"></parameter>
      <parameter name="Database User" defaultValue="__SQLUser__"></parameter>
      <parameter name="Database Password" defaultValue="__SQLPassword__"></parameter>
    </parameters>
  7. If I publish again, this time the new variables also appear in the setparameters .xml file
  8. Now I need to supress the auto generated creation of the connection string  parameter, and replace it with a parameter that uses the other parameters to generate the connection string. You would think this was a case of added more text to the parameters.xml file, but that does not work. If you add the block you would expect (making sure the name matches the auto generated connection string name) as below
    <parameter 
      defaultValue="Server=tcp:{Database Server}.database.windows.net,1433;Database={Database Name};User ID={Database User}@{Database Server};Password={Database Password};Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"
      description="Enter the value for FabrikamFiber-Express connection string"
      name="FabrikamFiber-Express-Web.config Connection String"
      tags="">
      <parameterentry
        kind="XmlFile"
        match="/configuration/connectionStrings/add[@name='FabrikamFiber-Express']/@connectionString"
        scope="\\web.config$" />
    </parameter>

    It does add the entry to setparameters.xml, but this blocks the successful operations at deployment. It seems that if a value needs to be generated from other variables there can be no entry for it in the setparameters.xml. Documentation hints you can set the Tag to ‘Hidden’ but this does not appear to work.

    One option would be to let the setparameters.xml file be generated and then remove the offending line prior to deployment but this feels wrong and prone to human error
  9. To get around this you need to added a file name <projectname>.wpp.target to the same folder as the project (and add it to the project). In this file place the following
    <?xml version="1.0" encoding="utf-8"?>
    <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Target Name="DeclareCustomParameters"
              BeforeTargets="Package">
        <ItemGroup>
          <MsDeployDeclareParameters Include="FabrikamFiber-Express">
            <Kind>XmlFile</Kind>
            <Scope>Web.config</Scope>
            <Match>/configuration/connectionStrings/add[@name='FabrikamFiber-Express']/@connectionString</Match>
            <Description>Enter the value for FabrikamFiber-Express connection string</Description>
            <DefaultValue>Server=tcp:{Database Server}.database.windows.net,1433;Database={Database Name};User ID={Database User}@{Database Server};Password={Database Password};Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;</DefaultValue>
            <Tags></Tags>
            <ExcludeFromSetParameter>True</ExcludeFromSetParameter>
          </MsDeployDeclareParameters>
        </ItemGroup>
      </Target>
      <PropertyGroup>
        <AutoParameterizationWebConfigConnectionStrings>false</AutoParameterizationWebConfigConnectionStrings>
      </PropertyGroup>
    </Project>

    The first block declares the parameter I wish to use to build the connection string. Note the ‘ExcludeFromSetParameter’ setting so this parameter is not in the setparameters.xml file. This is what you cannot set in the parameters.xml

    The second block stops the auto generation of the connection string. (Thanks to Sayed Ibrahim Hashimi for various posts on getting this working)
  10. Once the edits are made unload and reload the project as the <project>. wpp.targets file is cached on loading by Visual Studio.
  11. Make sure the publish profile is not set to generate a connection string

    image
  12. Now when you publish the project, you should get a setparameters.xml file with only the four  SQL variables, the AppSettings variables and the site name.
    (Note I have set the values for all of these to the format  __NAME__, this is so I can use token replacement in  my release pipeline)
    <?xml version="1.0" encoding="utf-8"?>
    <parameters>
      <setParameter name="IIS Web Application Name" value="__Sitename__" />
      <setParameter name="Location" value="__LOCATION__" />
      <setParameter name="Database Server" value="__sqlservername__" />
      <setParameter name="Database Name" value="__databasename__" />
      <setParameter name="Database User" value="__SQLUser__" />
      <setParameter name="Database Password" value="__SQLPassword__" />
    </parameters>
  13. If you deploy the web site, the web.config should have your values from the setparameters.xml file in it
    <appSettings>
       <add key="Location" value="__LOCATION__" />
    </appSettings>
    <connectionStrings>
         <add name="FabrikamFiber-Express" connectionString="Server=tcp:__sqlservername__.database.windows.net,1433;Database=__databasename__;User ID=__SQLUser__@__sqlservername__;Password=__SQLPassword__;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;" providerName="System.Data.SqlClient" />
    </connectionStrings>

You are now in a position manage the values of the setparameters.xml file however you wish. My choice is to use the ‘Replace Tokens’ build/release tasks from Colin’s ALM Corner Build & Release Tools Extension, as this tasks correctly handles secure/encrypted RM variables as long as you use the ‘Secret Tokens’ option on the advanced menu.

image



 

Summary

So yes, it all seems a but too complex, but it does work, and I think it makes for a cleaner deployment solution, less prone to human error. Which is what any DevOps solution must always strive for.

Depending on the values you put in the <project>.wpp.targets you can parameterise the connection string however you need.