Release Management components fail to deploy with a timeout if a variable is changed from standard to encrypted

I have been using Release Management to update some of our internal deployment processes. This has included changing the way we roll out MSDeploy packages; I am following Colin Dembovsky’s excellent post of the subject.

I hit an interesting issue today. One of the configuration variable parameters I was passing into a component was a password field. For my initial tests had just let this be a clear text ‘standard’ string in the Release Management. Once I got this all working I thought I better switch this variable to ‘encrypted’, so I just change the type on the Configuration Variables tab.

image 

On doing this I was warned that previous deployment would not be re-deployable, but that was OK for me, it was just a trial system. I would not be going back to older versions.

However when I tried to run this revised release template all the steps up to the edited MSDeploy step were fine, but the MSDeploy step never ran it just timed out. The component was never deployed to the target machine %appdata%\local\temp\releasemanagement folder.

image

In the end, after a few reboots to confirm the comms were OK, I just re-added the component to the release template and entered all the variables again. It then deployed without a problem.

I think this is a case of a misleading error message.

What I learnt getting Release Management running with a network Isolated environment

Updated 20 Oct 2014 – With notes on using an Action for cross domain authentication

In my previous post I described how to get a network isolated environment up and running with Release Management, it is all to do with shadow accounts. Well getting it running is one thing, having a useful release process is another.

For my test environment I needed to get three things deployed and tested

  • A SQL DB deployed via a DACPAC
  • A WCF web service deployed using MSDeploy
  • A web site deployed using MSDeploy

My environment was a four VM network isolated environment running on our TFS Lab Management system.

image 

The roles of the VMs were

  • A domain controller
  • A SQL 2008R2 server  (Release Management deployment agent installed)
  • A VM configured as a generic IIS web server (Release Management deployment agent installed)
  • A VM configured as an SP2010 server (needed in the future, but its presence caused me issues so I will mention it)

Accessing domain shares

The first issue we encountered was that we need the deployment agent on the VMs to be able to access domain shares on our corporate network, not just ones on the local network isolated domain. They need to be able to do this to download the actual deployment media. The easiest way I found to do this was to place a NET USE command at the start of the workflow for each VM I was deploying too. This allowed authentication from the test domain to our corporate domain and hence access for the agent to get the files it needed. The alternatives would have been using more shadow accounts, or cross domain trusts, both things I did not want the hassle of managing.

image

The run command line activity runs  the net command with the arguments use \\store\dropsshare [password] /user:[corpdomain\account]

I needed to use this command on each VM I was running the deployment agent on, so appears twice in this workflow, once for the DB server and once for the web server.

Updated 20 Oct 2014: After using this technique in a few release I realised it was a far better idea to have a action to do the job. The technique I mentioned above meant the password was in clear text, a parameterised action allows it to be encrypted.

To create an action (and it can be an action, not a component as it does not need to know the build location) needs the following settings

image

Version of SSDT SQL tools

My SQL instance was SQL 2008R2, when I tried to use the standard Release Management DACPAC Database Deployer tool it failed with assembly load errors. Basically the assemblies downloaded as part of the tool deployment did not match anything on the VM.

My first step was to install the latest SQL 2012 SSDT tools on the SQL VM. This did not help the problem as there was still a mismatch between the assemblies. I therefore create a new tool in the Release Management inventory, this was a copy of the existing DACPAC tool command, but using the current version of the tool assemblies from SSDT 2012

image

Using this version of the tools worked, my DB could be deployed/updated.

Granting Rights for SQL

Using SSDT to deploy a DB (especially if you have the package set to drop the DB) does  not grant any user access rights.

I found the easiest way to grant the rights the web service AppPool accounts needed was to run a SQL script. I did this by creating a component for my release with a small block of SQL to create DB owners, this is the same technique as used for the standard SQL create/drop activities shipped in the box with Release Management.

The arguments I used for the sqlcmd were -S __ServerName__ -b -Q “use __DBname__ ; create user [__username__] for login [__username__];  exec sp_addrolemember ‘db_owner’, ‘__username__’;”

image

Once I had created this component I could pass the parameters I needed add DB owners.

Creating the web sites

This was straight forward, I just used the standard components to create the required AppPools and the web sites. It is worth nothing that these command can be run against existing site, the don’t error if the site/AppPool already exists. This seems to be the standard model with Release Management as there is no decision (if) branching in the workflow, so all tools have to either work or stop the deployment.

image

I then used the irmsdeploy.exe Release Management component to run the MSDeploy publish on each web site/service

image

A note here: you do need make sure you set the path to the package to be the actual folder the .ZIP file is in, not the parental drop folder (in my case Lab\_PublishedWebsites\SABSTestHarness_Package not Lab)

image

Running some integration tests

We now had a deployment that worked. It pulled the files from our corporate LAN and deployed them into a network isolated lab environment.

I now wanted to run some tests to validate the deployment. I chose to use some SQL based tests that were run via MSTest. These tests had already been added to Microsoft Test Manager (MTM) using TCM, so I thought I had all I needed.

I added the Release Management MTM component to my workflow and set the values taken from MTM for test plan and suite etc.

image

However I quickly hit cross domain authentication issues again. The Release Management component does all this test management via a PowerShell script that runs TCM. This must communicate with TFS, which in my system was in the other domain, so fails.

The answer was to modify the PowerShell script to also pass some login credentials

image

The only change in the PowerShell script was that each time the TCM command is called the /login:$LoginCreds block is added, where $LoginCreds are the credentials passed in the form corpdomain\user,password

$testRunId = & “$tcmExe” run /create /title:”$Title” /login:$LoginCreds /planid:$PlanId /suiteid:$SuiteId /configid:$ConfigId /collection:”$Collection” /teamproject:”$TeamProject” $testEnvironmentParameter $buildDirectoryParameter $buildDefinitionParameter $buildNumberParameter $settingsNameParameter $includeParameter
   

An interesting side note is that if you try to run the TCM command at the command prompt you only need to provide the credentials on the first time it is run, they are cached. This does not seem to be the case inside the Release Management script, TCM is run three times, each time you need to pass the credentials.

Once this was in place, and suitable credentials added to the workflow I expected my test to run. They did but 50% failed – Why?

It runs out the issue was that in my Lab Management environment setup I had set the roles of both IIS server and SharePoint server to Web Server.

My automated test plan in MTM was set to run automated tests on the Web Server role, so sent 50% of the tests to each of the available servers. The tests were run by Lab Agent (not the deployment agent) which was running as the Network Service machine accounts e.g. Proj\ProjIIS75$ and Proj\ProjSp2010$. Only for former of these had been granted access to the SQL DB (it was the account being used for the AppPool), hence half the test failed, with DB access issues

I had two options here, grant both machine accounts access, or alter my Lab Environment. I chose the latter. I put the two boxes in different roles

image

I then had to load the test plan in MTM so it was updated with the changes

image

Once this was done my tests then ran as expected.

Summary

So I now have a Release Management deployment plan that works for a network isolated environment. I can run integration tests, and will soon add some CodeUI ones, it is should only be a case of editing the test plan.

It is an interesting question of how well Release Management, in its current form, works with Lab Management when it is SCVMM/Network Isolated environment based, is is certainly not its primary use case, but it can be done as this post shows. It certainly provides more options than the TFS Lab Management build template we used to use, and does provide an easy way to extend the process to manage deployment to production.

Getting started with Release Management with network isolated Lab Management environments

Our testing environments are based on TFS Lab Management, historically we have managed deployment into them manually (or at least via a PowerShell script run manually) or using TFS Build. I thought it time I at least tried to move over to Release Management.

The process to install the components of Release Management is fairly straight forward, there are wizards that ask little other than which account to run as

  • Install the deployment server, pointing at a SQL instance
  • Install the management client, pointing at the deployment server
  • Install the deployment agent on each box you wish to deploy to, again pointing it as the deployment server

I hit a problem with the third step. Our lab environments are usually network isolated, hence each can potentially be running their own copies of the same domain. This means the connection from the deployment agent to the deployment server is cross domain. We don’t want to setup cross domain trusts as

  1. we don’t want cross domain trusts, they are a pain to manage
  2. as we have multiple copies of environments there are more than one copy of some domains – all very confusing for cross domain trusts

So this means you have to use shadow accounts, as detailed in MSDN for Release Management, the key with this process is to make sure you manually add the accounts in Release Management (step 2) – I missed this at first as it differs from what you usually need to do.

To resolve this issue, add the Service User accounts in Release Management. To do this, follow these steps:

    1. Create a Service User account for each deployment agent in Release Management. For example, create the following:

      Server1\LocalAccount1
      Server2\LocalAccount1
      Server3\LocalAccount1

    2. Create an account in Release Management, and then assign to that account the Service User and Release Manager user rights. For example, create Release_Management_server\LocalAccount1.
    3. Run the deployment agent configuration on each deployment computer.

However I still had a problem, I entered the correct details in the deployment  configuration client, but got the error

image

The logs showed

Received Exception : Microsoft.TeamFoundation.Release.CommonConfiguration.ConfigurationException: Failed to validate Release Management Server for Team Foundation Server 2013.
   at Microsoft.TeamFoundation.Release.CommonConfiguration.DeployerConfigurationManager.ValidateServerUrl()
   at Microsoft.TeamFoundation.Release.CommonConfiguration.DeployerConfigurationManager.ValidateAndConfigure(DeployerConfigUpdatePack updatePack, DelegateStatusUpdate statusListener)
   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

A quick look using WireShark showed it was try to access http://releasemgmt.mydomain.com:1000/configurationservice.asmx,if I tried to access this in a browser it showed

 

Getting

Server Error in ‘/’ Application.
——————————————————————————–

Request format is unrecognized.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Web.HttpException: Request format is unrecognized.

Turns out the issue was I needed to be run the deployment client configuration tool as the shadow user account, not as any other local administrator.

Once I did this the configuration worked and the management console could scan for the client. So now I can really start to play…