But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

VSO now available in a European Azure Data Center

I don’t normal do posts that are just re-posts of TFS announcements, it is much better to get the information first hand from the original post, but this one is significant for us in Europe…

Up to now there has been a barrier to adoption of VSO that the underlying data will be hosted in the USA. Now there are all the usual Azure Microsoft guarantees about data security, but this has not been enough for some clients for legal, regulatory or their own reasons. This has made VSO a non-starter for many European’s where it at first appears a great match.

As of today you can now choose to host your new VSO account in Europe (Amsterdam data center). It won’t remove everyone's worries of cloud hosting, but certainly is a major step in the right direction from a European point of view, addressing many regulatory barriers.

Unfortunately we will have to wait a few sprints to be able to migrate any existing VSO instances, but you can’t have everything in one go!

For the full details have a look at Brian Harry’s and Jamie Cool’s posts

Visual Studio crashes when trying to add an item to a TFS build workflow

There has for a long time been an issue that when you try to add a new activity to the toolbox when editing a TFS build workflow Visual Studio can crash. I have seen it many times and never got to the bottom of it. It seems to be machine specific, as one machine can work while another supposedly identical will fail, but I could never track down the issue.

Today I was on a machine that was failing, but …

But I found a workaround in a really old forum post. The workaround is to load the IDE from the command line with the /safemode flag

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe /safemode

Once you do this you can edit the contents of our toolbox with crashes, and also your template if you wish. The best part is that once you exit the IDE and reload it as normal your new toolbox contents are still there.

Not perfect, but a good workaround

"The handshake failed due to an unexpected packet format" with Release Management

Whilst configuring a Release Management 2013.3 system I came across a confusing error. All seemed OK, the server, client and deployment agents were all installed and seemed to be working, but when I tried to select a build to deploy from both the Team Projects and Build drop downs were empty.

image

A check of the Windows event log on the server showed the errors

The underlying connection was closed: An unexpected error occurred on a send
The handshake failed due to an unexpected packet format

Turns out the issue was an incorrectly set value when the Release Management server was configured. HTTPS had been incorrectly selected, in fact there was no SSL certificate on the box so HTTPS could not work

image

As this had been done in error we did not use HTTPS at any other point in the installation. We always used the URL http://typhoontfs:1000 . The strange part of the problem was that the only time this mistake caused a problem was for the Team Project drop down, everything else seemed fine, clients and deployment agents all could see the server.

Once the Release Management server was reconfigured with the correct HTTP setting all was OK

image

Cannot build a SSRS project in TFS build due to expired license

If you want to get your TFS build process to product SSRS RDL files you need to call the vsDevEnv custom activity to run Visual Studio (just like for SSIS packages). On our new TFS2013.3 based build agents this step started to fail, turns out the issue was not incorrect versions of DLLs or a some badly applied update, but that the license for Visual Studio on the build agent had expire.

I found it by looking at diagnostic logs in the TFS build web UI.

image

To be able to build BI project with Visual Studio you do need a licensed copy of Visual Studio on the build agent. You can use a trial license, but it will expire. Also remember if you license VS by logging in with your MSDN Live ID that too needs to be refreshed from time to time (that is what go me), so better to use a product key.

Version stamping Windows 8 Store App manifests in TFS build

We have for a long time used the TFSVersion custom build activity to stamp all our TFS builds with a unique version number that matches out build number. However, this only edits the AssemblyInfo.cs file. As we are now building more and more Windows 8 Store Apps we also need to edit the XML in the Package.appxmanifest files used to build the packages too. Just like a Wix MSI project it is a good idea the package version matches some aspect of the assemblies it contains. We need to automate the update of this manifest as people too often forget to increment the version, causing confusion all down the line.

Now I could have written a new TFS custom activity to do the job, or edited the existing one, but both options seemed a poor choice. We all know that custom activity writing is awkward and a pain to support going forward. So I decided to use the hooks in the 2013 generation build process template to just call a custom PowerShell script to do the job.

I added a PreBuildScript.PS1 file as a solution item to my solution.

I placed the following code in the file. It uses the TFS environment variables to get the build location and version; using these to find and edit the manifest files. The only gotcha is files on the build box are read only (it is a server workspace) so the manifest file has to be set it to allow it to be written back too.

# get the build number, we assume the format is Myproject.Main.CI_1.0.0.18290
# where the version is set using the TFSVersion custom build activity (see other posts)

$buildnum = $env:TF_BUILD_BUILDNUMBER.Split('_')[1]
# get the manifest file paths
$files = Get-ChildItem -Path $env:TF_BUILD_BUILDDIRECTORY -Filter "Package.appxmanifest" -Recurse
foreach ($filepath in $files)
{
    Write-Host "Updating the Store App Package '$filepath' to version ' $buildnum '"
   # update the identity value
  
$XMLfile=NEW-OBJECT XML
    $XMLfile.Load($filepath.Fullname)
    $XMLFile.Package.Identity.Version=$buildnum
   # set the file as read write
    Set-ItemProperty $filepath.Fullname -name IsReadOnly -value $false
    $XMLFile.save($filepath.Fullname)
}

Note that any output sent via Write-Host will only appear in the diagnostic log of TFS. If you use Write-Error (or errors are thrown) these messages will appear in the build summary, but the build will not fail, but will be marked as a partial success.

Once this file was checked in i was able to reference the file in the build template

image

The build could not be run and got my Windows 8 Store packages with the required version number

Using MSDEPLOY from Release Management to deploy Azure web sites

Whilst developing our new set of websites we have been using MSDeploy to package up the websites for deployment to test and production Azure accounts. These deployments were being triggered directly using Visual Studio. Now we know this is not best practice, you don’t want developers shipping to production from their development PCs, so I have been getting around to migrating these projects to Release Management.

I wanted to minimise change, as we like MSDeploy, I just wanted to pass the extra parameters to allow a remote deployment as opposed to a local one using the built in WebDeploy component in Release Management

To do this I created a new component based on the WebDeploy tool. I then altered the arguments to

__WebAppName__.deploy.cmd /y /m:__PublishUrl__" -allowUntrusted /u:"__PublishUser__" /p:"__PublishPassword__" /a:Basic

image

With these three extra publish parameters I can target the deployment to an Azure instance, assuming WebDeploy is installed on the VM running the Release Management deployment client.

The required values for these parameters can be obtained from the .PublishSettings you download from your Azure web site’s management page. If you open this file in a text editor you can read the values need (bits you need are highlighted in yellow)

<publishData><publishProfile profileName="SomeSite - Web Deploy" publishMethod="MSDeploy" publishUrl="somesite.scm.azurewebsites.net:443" msdeploySite="SomeSite" userName="$SomeSite" userPWD="m1234567890abcdefghijklmnopqrstu" destinationAppUrl=http://somesite.azurewebsites.net SQLServerDBConnectionString="" mySQLDBConnectionString="" hostingProviderForumLink="" controlPanelLink="http://windows.azure.com"><databases/></publishProfile><publishProfile profileName="SomeSite- FTP" publishMethod="FTP" publishUrl=ftp://site.ftp.azurewebsites.windows.net/site/wwwroot ftpPassiveMode="True" userName="SomeSite\$SomeSite" userPWD=”m1234567890abcdefghijklmnopqrstu" destinationAppUrl=http://somesite.azurewebsites.net SQLServerDBConnectionString="" mySQLDBConnectionString="" hostingProviderForumLink="" controlPanelLink="http://windows.azure.com"><databases/></publishProfile></publishData>

These values are used as follows

  • WebAppName – This is the name of the MSDeploy package, this is exactly the same as a standard WebDeploy component.
  • PublishUrl - We need to add the https and the .axd to the start and end of the url e.g: https://somesite.scm.azurewebsites.net:443/MsDeploy.axd
  • PublishUser – e.g:  $SomeSite
  • PublishPassword – This is set as an encrypted parameter  so it cannot be viewed in the Release Management client e.g: m1234567890abcdefghijklmnopqrstu

On top of these parameters, we can still pass in extra parameters to transform the web.config using the setparameters.xml file as detailed in this other post  this allowing to complete the steps we need to do the configuration for the various environments in our pipeline.

“Communication with the deployer was lost during deployment” error with Release Management

Whilst developing  a new Release Management pipeline I did hit a problem that a component that published MSDeploy packages to Azure started to fail.  It had been working then suddenly I started seeing ‘communication with the deployer was lost during deployment’ messages as shown below.

image_thumb[1]

 

No errors were shown in any logs I could find, and no files appeared on the deployment target (you would expect the files/scripts to be deployed to appear on the machine running a RM Deployment client in the C:\users\[account]\local\temp\RM folder structure).

Rebooting of the Release Management server and deployment client had no effect.

The client in this case was a VM we use to do remote deployments e.g to Azure, SQL clusters etc. Place where we cannot install the RM Deployment Client. So it is used for a number of other pipelines, these were all working, so I doubted it was a communications issue.

In the end, out of frustration, I tried re-adding the component to the workflow and re-entering the parameters. Once this was done, and the old component instances delete, it all leapt into life.

I am not sure why I had the problems, I was trying to remember what I did exactly between the working and failing releases. All I can think is that I may have changed the password parameter to encrypted (I had forgotten to do this at first). Now I should have tried this sooner as I have posted on this error message before. All I can assume is that changing this parameter setting corrupted my component.

I should have read my own blog sooner

Publishing more than one Azure Cloud Service as part of a TFS build

Using the process in my previous post you can get a TFS build to create the .CSCFG and .CSPKG files needed to publish a Cloud Service. However, you hit a problem if your solution contains more that one Cloud Service project; as opposed to a single cloud service project with multiple roles, which is not a problem.

The method outlined in the previous post drops the two files into a Packages folder under the drops location. The .CSPKG files are fine, as they have unique names. However there is only one ServiceConfiguration.cscfg, whichever one was created last.

Looking in the cloud service projects I could find no way to rename the ServiceConfiguration file. It looks like it is like a app.config or web.config file i.e. it’s name is hard coded.

The only solution I could find was to add a custom target that is set to run after the publish target. This was added to the end of each .CCPROJ files using a text editor just before the closing </project>

 <Target Name="CustomPostPublishActions" AfterTargets="Publish">
    <Exec Command="IF '$(BuildingInsideVisualStudio)'=='true' exit 0
    echo Post-PUBLISH event: Active configuration is: $(ConfigurationName) renaming the .cscfg file to avoid name clashes
    echo Renaming the .CSCFG file to match the project name $(ProjectName).cscfg
    ren $(OutDir)Packages\ServiceConfiguration.*.cscfg $(ProjectName).cscfg
    " />
  </Target>
   <PropertyGroup>
    <PostBuildEvent>echo NOTE: This project has a post publish event</PostBuildEvent>
  </PropertyGroup>

 

Using this I now get unique name for the .CSCFG files as well as for .CSPKG files in my drops location. All ready for Release Management to pickup

Notes:

  • I echo out a message in the post build event too just as a reminder that I have added a custom target that cannot be seen in Visual Studio, so is hard to discover
  • I use an if test to make sure the commands are only run on the TFS build box, not on a local build. The main reason for this is the path names are different for local builds as opposed to TFS build. If you do want a rename on a local build you need to change the $(OutDir)Packages path to $(OutDir)app.publish. However, it seemed more sensible to leave the default behaviour occur when running locally

Getting the correct path and name for a project to pass as an MSBuild argument in TFS Build

I have been sorting out some builds for use with Release Management that include Azure Cloud Solutions. To get the correct packages built by TFS I have followed the process in my past blog post. The problem was I kept getting the build error

The target "Azure Packages\BlackMarble.Win8AppBuilder.AzureApi" does not exist in the project.

The issue was I could not get the solution folder/project name right for the MSBUILD target parameter. Was it the spaces in the folder? I just did not know.

The solution was to check the .PROJ file that was actually being run by MSBUILD. As you may know a .SLN file is not in MSBUILD format so you can’t just open it in notepad and look (unlike a .CSPROJ or .VBPROJ files), it is created by MSBUILD on the fly. To see this generated code, at a developer’s command prompt, run the following commands

cd c:\mysolutionroot
Set MSBuildEmitSolution=1
msbuild

When the MSBUILD command is run, whether the build works or not, there should be mysolution.sln.metaproj  file created. If you look in this file you will see the actual targets MSBUILD thinks it is dealing with.

In my case I could see

<Target Name="Azure Packages\BlackMarble_Win8AppBuilder_AzureApi:Publish">

So the first issue was my . were replaced by _

I changed my MSBUILD target argument to that shown in the file, but still had a problem. However, once I changed by space in the solution folder to %20 all was OK. So my final MSBUILD argument was

/t:Azure%20Packages\BlackMarble_Win8AppBuilder_AzureApi:Publish

image

Deploying a Windows service with Release Management

 recently needed to deploy a Windows service as part of a Release Management pipeline. In the past, our internal systems I have only need to deploy DB (via SSDT Dacpacs) and Websites (via MSDeploy), so a new experience.

WIX Contents

The first step to to create an MSI installer for the service. This was done using WIX, with all the fun that usually entails. The key part was a component to do the actual registration and starting of the service

<Component Id ="ModuleHostInstall" Guid="{3DF13451-6A04-4B62-AFCB-731A572C12C9}" Win64="yes">
   <CreateFolder />
   <Util:User Id="ModuleHostServiceUser" CreateUser="no" Name="[SERVICEUSER]" Password="[PASSWORD]" LogonAsService="yes" />
   <File Id="CandyModuleHostService" Name ="DataFeed.ModuleHost.exe" Source="$(var.ModuleHost.TargetDir)\ModuleHost.exe" KeyPath="yes" Vital="yes"/>
   <ServiceInstall Id="CandyModuleHostService" Name ="ModuleHost" DisplayName="Candy Module Host" Start="auto" ErrorControl="normal" Type="ownProcess"  Account="[SERVICEUSER]" Password="[PASSWORD]" Description="Manages the deployment of Candy modules" />
   <ServiceControl Id="CandyModuleHostServiceControl" Name="ModuleHost" Start="install" Stop="both" Wait="yes" Remove="uninstall"/>

So nothing that special here, but worth remembering if you miss out the ServiceControl block the service will not automatically start or be uninstalled with the MSI’s uninstall

You can see that we pass in the service account to be used to run the service as a property. This is an important technique for using WIX with Release Management, you will want to be able to pass in anything you may want to change as installation time as a parameter. This means we ended up with a good few properties such as

  <Property Id="DBSERVER" Value=".\sqlexpress" />
  <Property Id="DBNAME" Value ="=CandyDB" />
  <Property Id="SERVICEUSER" Value="Domain\serviceuser" />
  <Property Id="PASSWORD" Value="Password1" />

These tended to equate to app.config settings. In all cases I tried to set sensible default values so in most cases I could avoid passing in an override value.

These property values were then used to re-write the app.config file after the copying of the files from the MSI onto the target server. This was done using the XMLFile tools and some XPath e.g.

<Util:XmlFile Id="CacheDatabaseName" 
Action="setValue"
Permanent="yes"
File="[#ModuleHost.exe.config]"
ElementPath="/configuration/applicationSettings/DataFeed.Properties.Settings/setting[\[]@name='CacheDatabaseName'[\]]/value" Value="[CACHEDATABASENAME]" Sequence="1" />
 

Command Line Testing

Once the MSI was built it could be tested from the command line using the form

msiexec /i Installer.msi /Lv msi.log SERVICEUSER="domain\svc_acc" PASSWORD="Password1" DBSERVER="dbserver" DBSERVER="myDB" …..

I soon spotted a problem. As I was equating properties with app.config settings I was passing in connections strings and URLs, so the command line got long very quickly. It was really unwieldy to handle

A check of the log file I was creating, msi.log, showed the command line seemed to be truncated. This seemed to occur around 1000 characters. I am not sure if this was an artefact of the logging or the command line, but either way a good reason to try to shorten the property list.

I  therefore decided that I would not pass in whole connection strings, but just the properties that might change, especially effective for connection strings to things such as Entity Framework. This meant I did some string building in WIX during the transformation of the app.config file e.g.

<Util:XmlFile Id='CandyManagementEntities1'
   Action='setValue'
   ElementPath='/configuration/connectionStrings/add[\[]@name="MyManagementEntities"[\]]/@connectionString'
   File='[#ModuleHost.exe.config]' Value='metadata=res://*/MyEntities.csdl|res://*/MyEntities.ssdl|res://*/MyEntities.msl;provider=System.Data.SqlClient;provider connection string=&quot;data source=[DBSERVER];initial catalog=[DBNAME];integrated security=True;MultipleActiveResultSets=True;App=EntityFramework&quot;' />

This technique had another couple of advantages

  • It meant I did not need to worry over spaces in strings, I could therefore lose the “ in the command line – Turns out this is really important later.
  • As I was passing in just a ‘secret value’ as opposed to a whole URL I could use the encryption features of Release Management to hide certain values

It is at this point I was delayed for a long time. You have to be really careful when installing Windows services via an MSI that your service can actually start. If it cannot then you will get errors saying "… could not be installed. Verify that you have sufficient privileges to install system services". This is probably not really a rights issue, just that some configuration setting is wrong so the service has failed to start. In my case it was down to an incorrect connection string, stray commas and quotes, and a missing DLL that should have been in the installer. You often end up working fairly blind at this point as Windows services don’t give too much information when they fail to load. Persistence, SysInternals Tools and comparing to the settings/files on a working development PC are the best options

Release Management Component

Once I had working command line I could create a component in Release Management. On the Configure Apps > Components page I already had a MDI Deployer, but this did not expose any properties. I therefore copied this component to create a MSI deployer specific to my new service installer and started to edit it.

All the edits were on the deployment tab, adding the extra properties that could be configured.

image

Note: Now it might be possible to do something with the pre/post deployment configuration variables as we do with MSDeploy, allowing the MSI to run then editing the app.config later. However, given that MSI service installers tends to fail they cannot start the new service I think passing in the correct properties into MSIEXEC is a better option. Also means it is consistent for anyone using the MSI via the command line.

On the Deployment tab I changed the Arguments to

-File ./msiexec.ps1 -MsiFileName "__Installer__"  -MsiCustomArgs ‘SERVICEUSER=”__SERVICEUSER__”  PASSWORD=”__PASSWORD__” DBSERVER=”__DBSERVER__”  DBNAME=”__DBNAME__” …. ’

I had initially assumed I needed the quotes around property values. Turns out I didn’t, and due to the way Release Management runs the component they made matters much, much worse. MSIEXEC kept failing instantly. if I ran the command line by hand on the target machine it was actually showing the Help dialog, so I knew the command line was invalid.

Turns out the issue is Release Management calls PowerShell.EXE to run the script passing in the Arguments. This in turn calls a PowerShell Script which does some argument processing before running a process to run MSIEXEC.EXE with some parameters. You can see there are loads of places where the escaping and quotes around parameters could get confused.

After much fiddling, swapping ‘ for “ I realised I could just forget most of the quotes. I had already edited my WIX package to build complex strings, so the actual values were simple with no spaces. Hence my command line became

-File ./msiexec.ps1 -MsiFileName "__Installer__"  -MsiCustomArgs “SERVICEUSER=__SERVICEUSER__  PASSWORD=__PASSWORD__ DBSERVER=__DBSERVER__  DBNAME=__DBNAME__ …. “

Once this was set my release pipeline worked resulting in a system with DBs, web services and window service all up and running.

As is often the case it took a while to get this first MSI running, but I am sure the next one will be much easier.