But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

vNext Release Management and Network Isolation

If you are trying to use Release Management or any deployment tool with a network isolated Lab Management setup you will have authentication issues. Your isolated domain is not part of your production domain, so you have to provide credentials. In the past this meant Shadow Accounts or the simple expedient of  running a NET USE at the start of your deployment script to provide a login to the drops location.

In Release Management 2013.4 we get a new option to address this issue if you are using DSC based deployment. This is Deploy from a build drop using a shared UNC path. In this model the Release Management server copies the contents of the drops folder to a known share and passes credentials to access it down to the DSC client (you set these as parameters on the server).

This is a I nice formalisation of the tricks we had to pull by hand in the past. And something I had missed when the Update 4 came out

Thoughts in vNext deployment in Release Management

The DCS two step

When working with DSC a difficult concept can be that your desired state script is ‘compiled’ to a MOF file that is then ‘run’ by the desired state manager on the target machine. This is a two step affair and you have no real input on the second part. This is made more complex when Release Management is involved.

Colin Dembovsky did an excellent pair of posts on getting Release Management working with DSC based vNext templates. The core of the first post is that he made use of the Script DSC resource to run SQLPackage.EXE and MSDeploy to do the actual deployment of a system as well as using it to manage the transformation of the configuration files in his MSDeploy package.

This is where the two step issue raises it head. Even with his post I managed to get myself very confused.

The problem is Release Management passes variable into the DSC script and these are evaluated when the MOF is compiled. For most resources this is fine, but for the Script resource you have to be very aware that the string that is the actual script is treated as just a string, not code, so variables are not evaluated until the MOF is run by the desired state manager, by which point there are no Release Management variables set (they are long gone).

Colin’s provides an answer to this problem with his cScriptWithParams resource. This resource takes the Release Management provided properties and passes them into as parameters into the MOF compilation, forcing their evaluation, neatly side stepping the problem. He uses this  technique for the SetParameters.XML transform.

This is all good, but it got me thinking, his post has a number of hard coded paths, and also copies the deployment files to a ‘known location’. Is this really all required if we can pass in the Release Management $ApplicationPath?

So I swapped all my Script resources to use the cScriptWithParams resource passing in the applicationpath thus removing the need to copy the files from their default location.

      cScriptWithParams SetConStringDeployParam
        {
            GetScript = { @{ Name = "SetDeployParams" } }
            TestScript = { $false }
            SetScript = {
                $paramFilePath = "$folder\_PublishedWebsites\WcfService_Package\WcfService.SetParameters.xml"
 
                $paramsToReplace = @{
                      "__DBContext__" = $context
                      "__SiteName__" = $siteName
                }
 
                $content = gc $paramFilePath
                $paramsToReplace.GetEnumerator() | % {
                    $content = $content.Replace($_.Key, $_.Value)
                }
                sc -Path $paramFilePath -Value $content
            }
            cParams =
            @{
                context = $context;
                siteName = $siteName;
                folder = $ApplicationPath;

            }
        }

       
        cScriptWithParams DeploySite
        {
            GetScript = { @{ Name = "DeploySite" } }
            TestScript = { $false }
            SetScript = {
                & "$folder\_PublishedWebsites\WcfService_Package\WcfService.deploy.cmd" /Y
            }
           
            cParams =
            @{
                folder = $ApplicationPath;
            }

            DependsOn = "[cScriptWithParams]SetConStringDeployParam"
           
        }

I think this gave a easier to follow script, though I do wonder about my naming convention, maybe I need to adopt a nomenclature for inner script variables as opposed to global ones

Where are my parameters stored?

However, this does raise the question of where do these ‘global’ parameters come from? We have two options

  • A PowerShell Configuration data file (the standard DSC way)
  • Release management parameters

Either are valid, if you want to source control all  your configuration the first option is good. However what happens if you need to store secrets? In this case the ability to store a value encrypted within Release Management is useful.

In reality I expect will will use a combination. maybe with everything bar secrets in the configuration file.

Setting a build version in a JAR file from TFS build

Whilst helping a Java based team (part of larger organisation that used many sets of both Microsoft and non-Microsoft tools) to migrate from Subversion to TFS I had to tackle their Jenkins/Ant based builds.

They could have stayed on Jenkins and switched to the TFS source provider, but they wanted to at least look at how TFS build would better allow them to  trace their builds against TFS work items.

All went well, we setup a build controller and agent specifically for their team and installed Java onto it as well the TFS build extensions. We were very quickly able to get our test Java project building on the new build system.

One feature that their old Ant scripts used was to store the build name/number into the Manifest of any JAR files created, a good plan as it is always good to know where something came from.

When asked as how to do this with TFS build I thought ‘no problem I will just use TFS build environment variable’ and add something like the following

<property environment="env"/>

<target name="jar">
        <jar destfile="${basedir}/javasample.jar" basedir="${basedir}/bin">
            <manifest>
                <attribute name="Implementation-Version" value="${env.TF_BUILD_BUILDNUMBER}" />
            </manifest>   
        </jar>
</target>

But this did not work, I just saw the text ${env.TF_BUILD_BUILDNUMBER}" in my manifest, basically the environment variable could not be resolved.

After a bit more of think I realised the problem is that the Ant/Maven build extensions for TFS are based on TFS 2008 style builds, the build environment variables are a TFS 2012 and later feature, so of course they are not set.

A quick look in the automatically generated TFSBuild.proj file generated for the build showed that the MSBuild $(BuildNumber) was passed into the Ant script as a property, so it could be referenced in the Ant Jar target (note the brackets change from () to {})

<target name="jar">
        <jar destfile="${basedir}/javasmaple.jar" basedir="${basedir}/bin">
            <manifest>
                <attribute name="Implementation-Version" value="${BuildNumber}" />
            </manifest>   
        </jar>
</target>

 

Once this change was made I then got the manifest I expected including the build number

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.4
Created-By: 1.8.0_25-b18 (Oracle Corporation)
Implementation-Version: JavaSample.Ant.Manual_20141216.7

Can’t build SSDT projects in a TFS build

Whilst building a new TFS build agent VM using our standard scripts I hit a problem that SSDT projects would not build, but was fine on our existing agents. The error was

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets (513): The "SqlBuildTask" task failed unexpectedly.
System.MethodAccessException: Attempt by method 'Microsoft.Data.Tools.Schema.Sql.Build.SqlTaskHost.OnCreateCustomSchemaData(System.String, System.Collections.Generic.Dictionary`2<System.String,System.String>)' to access method 'Microsoft.Data.Tools.Components.Diagnostics.SqlTracer.ShouldTrace(System.Diagnostics.TraceEventType)' failed.

The problem was fixed by doing an update via the Visual Studio > Tools > Extensions and Updates. Once this was completed the build was fine.

Seems there may have been an issue with the Update 3 generation SSDT tools, so older and newer versions seem OK. Our existing agents had already been patched.

TFS announcements roundup

There have been a load on announcements about TFS, VSO and Visual Studio in general in the past couple of week, mostly at the Connect() event.

Just to touch on a few items

If you have not had a chance to have look at these features try the videos of all the sessions on Channel9, the keynotes are a good place to start. Also look, as usual, at the various posts on Brian Harry’s Blog. It is time of rapid change in ALM tooling

Errors running tests via TCM as part of a Release Management pipeline

Whilst getting integration tests running as part of a Release Management  pipeline within Lab Management I hit a problem that TCM triggered tests failed as the tool claimed it could not access the TFS build drops location, and that no .TRX (test results) were being produced. This was strange as it used to work (the RM system had worked when it was 2013.2, seems to have started to be issue with 2013.3 and 2013.4, but this might be a coincidence)

The issue was two fold..

Permissions/Path Problems accessing the build drops location

The build drops location passed is passed into the component using the argument $(PackageLocation). This is pulled from the component properties, it is the TFS provided build drop with a \ appended on the end.

image 

Note that the \ in the text box is there as the textbox cannot be empty. It tells the component to uses the root of the drops location. This is the issue, as when you are in a network isolated environment and had to use NET USE to authenticate with a the TFS drops share the trailing \ causes a permissions error (might occur in other scenarios too I have not tested it).

Removing the slash or adding a . (period) after the \ fixes the path issue, so..

  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779        -  works
  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779\      - fails 
  • \\server\Drops\Services.Release\Services.Release_1.0.227.19779\.     - works 

So the answer is add a . (period) in the pipeline workflow component so the build location is $(PackageLocation). as opposed to $(PackageLocation) or to edit the PS1 file that is run to do some validation to strip out any trailing characters. I chose the later, making the edit

if ([string]::IsNullOrEmpty($BuildDirectory))
    {
        $buildDirectoryParameter = [string]::Empty
    } else
    {
        # make sure we remove any trailing slashes as the cause permission issues
        $BuildDirectory = $BuildDirectory.Trim()
        while ($BuildDirectory.EndsWith("\"))
        {
            $BuildDirectory = $BuildDirectory.Substring(0,$BuildDirectory.Length-1)
        }
        $buildDirectoryParameter = "/builddir:""$BuildDirectory"""
    }
   

Cannot find the TRX file even though it is present

Once the tests were running I still had an issue that even though TCM had run the tests, produced a .TRX file and published it’s contents back to TFS, the script claimed the file did not exist and so could not pass the test results back to Release Management.

The issue was the call being used to check for the file existence.

[System.IO.File]::Exists($testRunResultsTrxFileName)

As soon as I swapped to the recommended PowerShell way to check for files

Test-Path($testRunResultsTrxFileName)

it all worked.

‘Test run must be created with at least one test case’ error when using TCM

I have been setting up some integration tests as part of a release pipeline. I am using TCM.EXE to trigger tests from the command line. Something along the lines

TCM.exe run /create /title:"EventTests" /collection:"http://myserver:8080/tfs” /teamproject:myteamproject /testenvironment:"Integration" /builddir:\\server\Drops\Build_1.0.226.1975”  /include /planid:26989  /suiteid:27190 /configid:1

I kept getting the error

‘A test run must be created with at least one test case’

Strange thing was my test suite did contains a number of test, and they were marked as active.

The issue was actually the configid it was wrong, there is no easy way to check them from the UI. use the following command to get a list of valid IDs

TCM.exe configs /list   /collection:"http://myserver:8080/tfs” /teamproject:myteamproject

Id        Name
--------- ----------------------------------------------------------------
35        Windows 8.1 ARM
36        Windows 8.1 64bit
37        Windows 8.1 ATOM
38        Default configuration created @ 11/03/2014 12:58:15
39        Windows Phone 8.1

Your can now use the correct ID, not one you had to guess

Linking VSO to your Azure Subscription and Azure Active Directory

I have a few old Visual Studio Online (VSO) accounts (dating back to TFSPreview.com days). We use them to collaborate with third parties, it was long overdue that I tidied them up; as a problem historically has been that all access to VSO has been using a Microsoft Accounts (LiveID, MSA), these are hard to police, especially if users mix personal and business ones.

The solution is to link your VSO instance to an Azure Active Directory (AAD). This means that only users listed in the AAD can connect to the VSO instance. As this AAD can be federated to an on-prem company AD it means that the VSO users can be either

  • Company domain users
  • MSA accounts specifically added to AAD

Either way it gives the AAD administrator an easy way to manage access to VSO. A user with a MSA, even if an administrator in VSO cannot add any unknown users to VSO. For details see MSDN. All straight forward you would think, but it I had a few issues.

The problem was I had setup my VSO accounts using a MSA in the form user@mycompany.co.uk, this was also linked to my MSDN subscription.  As part of the VSO/AAD linking process I needed to add the MSA user@mycompany.co.uk to our AAD, but I could not. The AAD was setup for federation of accounts in the mycompany.com domain, so you would have thought I would be OK, but back in our on-prem AD (the one it was federated to) I had  user@mycompany.co.uk as an email alias for user@mycompany.com. Thus blocked the adding of the user to AAD, hence I could got link VSO to Azure.

The answer was to

  1. Add another MSA account to the VSO instance, one unknown to our AD even as an alias e.g. user@live.co.uk 
  2. Make this user the owner of the VSO instance.
  3. Add the user@live.co.uk MSA to the AAD directory
  4. Make them an Azure Subscription administrator.
  5. Login to the Azure portal as this MSA, once this was done the VSO could be linked to the AAD directory.
  6. I could then make an AAD user (user@mycompany.com) a VSO user and then the VSO owner
  7. The user@live.co.uk MSA could then be deleted from VSO and AAD
  8. I could then login to VSO as  my user@mycompany.com AAD account, as opposed to the old user@mycompany.co.uk MSA account

Simple wasn’t it!

We still had one problem, and that was user@mycompany.com was showing as a basic user in VSO, if you tried to set it to MSDN eligible flipped back to basic.

The problem here was we had not associated the AAD account user@mycompany.com with the MSA account user@mycompany.co.uk in the MSDN portal (see MSDN).

Once this was done it all worked as expected, VSO picking up that my AAD account had a full MSDN subscription.

Cannot see a TFS drops location from inside a network isolated environment for Release Management

I have posted before about using Release Management with Lab Management network isolation. They key is that you must issue a NET USE command at the start of the pipeline to allow the VMs in the isolated environment the ability to see the TFS drops location.

I hit a problem today that a build failed even though I had issued the NET USE.

I got the error

Package location '\\store\drops\Sabs.Main.CI\Sabs.Main.CI_2.5.178.19130\\' does not exist or Deployer user does not have access.

Turns out the problem was I had issued the NET USE as \\store.blackmarble.co.uk\drops not \\store\drops it is vital the names match up. Once I changed this my NET USE all was OK

VSO now available in a European Azure Data Center

I don’t normal do posts that are just re-posts of TFS announcements, it is much better to get the information first hand from the original post, but this one is significant for us in Europe…

Up to now there has been a barrier to adoption of VSO that the underlying data will be hosted in the USA. Now there are all the usual Azure Microsoft guarantees about data security, but this has not been enough for some clients for legal, regulatory or their own reasons. This has made VSO a non-starter for many European’s where it at first appears a great match.

As of today you can now choose to host your new VSO account in Europe (Amsterdam data center). It won’t remove everyone's worries of cloud hosting, but certainly is a major step in the right direction from a European point of view, addressing many regulatory barriers.

Unfortunately we will have to wait a few sprints to be able to migrate any existing VSO instances, but you can’t have everything in one go!

For the full details have a look at Brian Harry’s and Jamie Cool’s posts