But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

An alternative to setting a build quality on a TFS vNext build

TFS vNext builds do not have a concept of build quality unlike the old XAML based builds. This is an issue for us as we used the changing of the build quality as signal to test a build, or to mark it as released to a client (this was all managed with my TFS Alerts DSL to make sure suitable emails and build retention were used).

So how to get around this problem with vNext?

I have used Tag on builds, set using the same REST API style calls as detailed in my post on Release Management vNext templates. I also use the REST API to set the retention on the build, so I actually now don’t need to manage this via the alerts DSL.

The following script, if used to wrapper the calling of integration tests via TCM, should set the tags and retention on a build


function Get-BuildDetailsByNumber
{
    param
    (
        $tfsUri ,
        $buildNumber,
        $username,
        $password

    )

    $uri = "$($tfsUri)/_apis/build/builds?api-version=2.0&buildnumber=$buildNumber"

    $wc = New-Object System.Net.WebClient
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
    write-verbose "Getting ID of $buildNumber from $tfsUri "

    $jsondata = $wc.DownloadString($uri) | ConvertFrom-Json
    $jsondata.value[0]
 
}

function Set-BuildTag
{
    param
    (
        $tfsUri ,
        $buildID,
        $tag,
        $username,
        $password

    )

 
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
   
    write-verbose "Setting BuildID $buildID with Tag $tag via $tfsUri "

    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)/tags/$($tag)?api-version=2.0"

    $data = @{value = $tag } | ConvertTo-Json

    $wc.UploadString($uri,"PUT", $data)
   
}

function Set-BuildRetension
{
    param
    (
        $tfsUri ,
        $buildID,
        $keepForever,
        $username,
        $password

    )

 
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
   
    write-verbose "Setting BuildID $buildID with retension set to $keepForever via $tfsUri "

    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)?api-version=2.0"
    $data = @{keepForever = $keepForever} | ConvertTo-Json
    $response = $wc.UploadString($uri,"PATCH", $data)
   
}


# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose

$ErrorActionPreference = 'Continue' # this controls if any test failure cause the script to stop

 

$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition

write-verbose "Running $folder\TcmExec.ps1"

 

& "$folder\TcmExec.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -SettingsName $SettingsName write-verbose "TCM exited with code '$LASTEXITCODE'"
$newquality = "Test Passed"
$tag = "Deployed to Lab"
$keep = $true
if ($LASTEXITCODE -gt 0 )
{
    $newquality = "Test Failed"
    $tag = "Lab Deployed failed"
    $keep = $false
}
write-verbose "Setting build tag to '$tag' for build $BuildNumber"


$url = "$Collection/$Teamproject"
$jsondata = Get-BuildDetailsByNumber -tfsUri $url -buildNumber $BuildNumber #-username $TestUserUid -password $TestUserPwd
$buildId = $jsondata.id
write-verbose "The build $BuildNumber has ID of $buildId"
 
write-verbose "The build tag set to '$tag' and retention set to '$key'"
Set-BuildTag -tfsUri $url  -buildID $buildId -tag $tag #-username $TestUserUid -password $TestUserPwd
Set-BuildRetension -tfsUri $url  -buildID $buildId  -keepForever $keep #-username $TestUserUid -password $TestUserPwd

# now fail the stage after we have sorted the logging
if ($LASTEXITCODE -gt 0 )
{
    Write-error "Test have failed"
}

If all the tests pass we see the Tag being added and the retention being set, if they fail just a tag should be set

image

$ErrorActionPreference = 'Continue'

Cannot create an MSDeploy package for an Azure Web Job project as part of an automated build/

I like web deploy as a means to package up websites for deployment. I like the way I only need to add

/p:DeployOnBuild=True;PublishProfile=Release

as an MSBuild argument to get the package produced as part of an automated build. This opening up loads of deployment options

I recently hit an issue packaging up a solution that contained an Azure WebSite and an Azure Web Job (to be hosted in the web site). It is easy to add the web job so that it is included in the Web Deploy package. Once this was done we could deploy from Visual Studio, or package to the local file system and see the web job EXE in the app_data\jobs folder as expected.

The problems occurred when we tried to get TFS build to create the deployment package using the arguments shown above. I got the error

The value for PublishProfile is set to 'Release', expected to find the file at 'C:\vNextBuild\_work\4253ff91\BM\Src\MyWebJob\Properties\PublishProfiles\Release.pubxml' but it could not be found.

The issue is that there is a Publish target for the web jobs project type, but if run from Visual Studio it actually creates a ClickOnce package. This wizard provides no means create an MSDeploy style package.

MSBuild is getting confused as it expects there to be this MSDeploy style package definition for the web job projects, even though it won’t actually use it as the Web Job EXE will be copied into the web site deployment package.

The solution was to add a dummy PublishProfiles\Release.pubxml file into the properties folder of the web jobs project.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>Package</WebPublishMethod>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SiteUrlToLaunchAfterPublish />
    <LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <DesktopBuildPackageLocation />
    <PackageAsSingleFile>true</PackageAsSingleFile>
    <DeployIisAppPath />
    <PublishDatabaseSettings/>
    </PropertyGroup>
</Project>

Note: I had to add this file to source control via the TFS Source Code Explorer as Visual Studio does not allow you add folders/files manually under the properties folder.

Once this file was added my automated build worked OK, and I got my web site package including the web job.

TF30063 Errors accessing a TFS 2015 server via the C# API after upgrade from 2013

Background

We  upgraded our production TFS 2013.4 server to TFS 2015 RTM this week. As opposed to an in-place upgrade we chose to make a few change on the way; so whilst leaving our DBs on our SQL 2012 cluster

  • We moved to a new VM for our AT (to upgrade from Windows 2008R2 to 2012R2)
  • Split the SSRS instance off the AT to a separate VM with a new SSAS server (again to move to 2012R2 and to ease management, getting all the reporting bits in one place)

But we do not touch

  • Our XAML Build systems leaving them at 2013 as we intend to migrate to vNext build ASAP
  • Our Test Controller/Release Management/Lab Environment leaving it at 2013 for now, as we have other projects on the go to update the hardware/cloud solutions underpinning theses.

All went well, no surprises, the running of the upgrade tool took about 1 hour.

The Problem

The only problem we have had was to do with my TFS Alerts DSL Processor, which listens for TFS Alerts and runs custom scripts . I host this on the TFS AT, and I would expect it to set build retention and send emails when a TFS XAML Build quality changes. This did not occur, in the Windows error log  I was seeing

2015-08-12 21:04:02.4195 ERROR TFSEventsProcessor.DslScriptService: TF30063: You are not authorized to access https://tfs.blackmarble.co.uk/tfs/DefaultCollection.

After much fiddling, including writing a small command line test client, I confirmed that the issue was specific to the production server. The tool ran fine on other PCs, but on the live server a Window authentication dialog was shown which would not accept any valid credentials

It was not as I had feared a change in the TFS API, in fact there is no reason my 2012 or 2013 API targeted version of the TFS Alert DSL should not be able to talk to a TFS 2015 server as long as the correct version of the TFS API is installed on the machine hosting the DSL.

The Solution

The issue was due to Windows loopback protection. This had been disabled on our old old TFS AT, but not on the new one. As we wanted to avoid changing the global loopback protection setting we set the following via Regedit to allow it for a single CName

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
    ValueName - BackConnectionHostNames
    Type - multistring
    Data  - tfs.blackmarble.co.uk

Once this was done(and without a reboot) my alerts processing work without any problems.

Running Microsoft Test Manager Test Suites as part of a vNext Release pipeline - Part 2

In my last post I discussed how you could wire TCM tests into a Release Management vNext pipeline. The problem with the script I provided, as I noted, was that the deployment was triggered synchronously by the build i.e. the build/release process was:

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Wait while the RM pipeline completed
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
  3. When RM completed the TFS build completes

This process raised a couple of problems

  • You cannot associate the integration tests with the build as TCM only allow association with completed successful builds. When TCM finishes in this model the build is still in progress.
  • You have to target only the first automated stage of the pipeline, else the build will be held as ‘in progress’ until all the release stages have complete, which may be days if there are manual approvals involved

The script InitiateReleaseFromBuild

These problems can all be fixed by altering the PowerShell that triggers the RM pipeline so that it does not wait for the deployment to complete, so the TFS build completes as soon as possible.

This is done by passing in an extra parameter which is set in TFS build

param(
    [string]$rmserver = $Args[0],
    [string]$port = $Args[1], 
    [string]$teamProject = $Args[2],  
    [string]$targetStageName = $Args[3],
    [string]$waitForCompletion = $Args[4]
)

cls
$teamFoundationServerUrl = $env:TF_BUILD_COLLECTIONURI
$buildDefinition = $env:TF_BUILD_BUILDDEFINITIONNAME
$buildNumber = $env:TF_BUILD_BUILDNUMBER


"Executing with the following parameters:`n"
"  RMserver Name: $rmserver"
"  Port number: $port"
"  Team Foundation Server URL: $teamFoundationServerUrl"
"  Team Project: $teamProject"
"  Build Definition: $buildDefinition"
"  Build Number: $buildNumber"
"  Target Stage Name: $targetStageName`n"
"  Wait for RM completion: $waitForCompletion`n"

$wait = [System.Convert]::ToBoolean($waitForCompletion)
$exitCode = 0

trap
{
  $e = $error[0].Exception
  $e.Message
  $e.StackTrace
  if ($exitCode -eq 0) { $exitCode = 1 }
}

$scriptName = $MyInvocation.MyCommand.Name
$scriptPath = Split-Path -Parent (Get-Variable MyInvocation -Scope Script).Value.MyCommand.Path

Push-Location $scriptPath   

$server = [System.Uri]::EscapeDataString($teamFoundationServerUrl)
$project = [System.Uri]::EscapeDataString($teamProject)
$definition = [System.Uri]::EscapeDataString($buildDefinition)
$build = [System.Uri]::EscapeDataString($buildNumber)
$targetStage = [System.Uri]::EscapeDataString($targetStageName)

$serverName = $rmserver + ":" + $port
$orchestratorService = "http://$serverName/account/releaseManagementService/_apis/releaseManagement/OrchestratorService"

$status = @{
    "2" = "InProgress";
    "3" = "Released";
    "4" = "Stopped";
    "5" = "Rejected";
    "6" = "Abandoned";
}

$uri = "$orchestratorService/InitiateReleaseFromBuild?teamFoundationServerUrl=$server&teamProject=$project&buildDefinition=$definition&buildNumber=$build&targetStageName=$targetStage"
"Executing the following API call:`n`n$uri"

$wc = New-Object System.Net.WebClient
$wc.UseDefaultCredentials = $true
# rmuser should be part rm users list and he should have permission to trigger the release.

#$wc.Credentials = new-object System.Net.NetworkCredential("rmuser", "rmuserpassword", "rmuserdomain")

try
{
    $releaseId = $wc.DownloadString($uri)

    $url = "$orchestratorService/ReleaseStatus?releaseId=$releaseId"

    $releaseStatus = $wc.DownloadString($url)


    if ($wait -eq $true)
    {
        Write-Host -NoNewline "`nReleasing ..."

        while($status[$releaseStatus] -eq "InProgress")
        {
            Start-Sleep -s 5
            $releaseStatus = $wc.DownloadString($url)
            Write-Host -NoNewline "."
        }

        " done.`n`nRelease completed with {0} status." -f $status[$releaseStatus]
    } else {

        Write-Host -NoNewline "`nTriggering Release and exiting"
    }

}
catch [System.Exception]
{
    if ($exitCode -eq 0) { $exitCode = 1 }
    Write-Host "`n$_`n" -ForegroundColor Red
}

if ($exitCode -eq 0)
{
    if ($wait -eq $true)
    {
        if ($releaseStatus -eq 3)
        {
          "`nThe script completed successfully. Product deployed without error`n"
        } else {
            Write-Host "`nThe script completed successfully. Product failed to deploy`n" -ForegroundColor Red
            $exitCode = -1 # reset the code to show the error
        }
    } else {
        "`nThe script completed successfully. Product deploying`n"
    }
}
else
{
  $err = "Exiting with error: " + $exitCode + "`n"
  Write-Host $err -ForegroundColor Red
}

Pop-Location

exit $exitCode

The Script TcmExecWrapper

A change is also required in the wrapper script I use to trigger the TCM test run. We need to check the exit code from the inner TCM PowerShell script and update the TFS build quality appropriately.

To this I use the new REST API in TFS 2015 as this is far easier than using the older .NET client API. No DLLs to distribute.

It is worth noticing that

  • I pass the credentials into the script from RM that are used to talk to the TFS server. This is because I am running my tests in a network isolated TFS Lab Environment, this means I am in the wrong domain to see the TFS server without providing login details. If you are not working cross domain you could just use Default Credentials.
  • RM only passes the BuildNumber into the script e.g. MyBuild_1.2.3.4, but the REST API need the build id to set the quality. Hence the need for function Get-BuildDetailsByNumber to get the id from the name
# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose
function Get-BuildDetailsByNumber
{
    param
    (
        $tfsUri ,
        $buildNumber,
        $username,
        $password
    )
    $uri = "$($tfsUri)/_apis/build/builds?api-version=2.0&buildnumber=$buildNumber"
    $wc = New-Object System.Net.WebClient
    #$wc.UseDefaultCredentials = $true
    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
   
    write-verbose "Getting ID of $buildNumber from $tfsUri "
    $jsondata = $wc.DownloadString($uri) | ConvertFrom-Json
    $jsondata.value[0]
 
}
function Set-BuildQuality
{
    param
    (
        $tfsUri ,
        $buildID,
        $quality,
        $username,
        $password
    )
    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)?api-version=1.0"
    $data = @{quality = $quality} | ConvertTo-Json
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    #$wc.UseDefaultCredentials = $true
    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
   
    write-verbose "Setting BuildID $buildID to quality $quality via $tfsUri "
    $wc.UploadString($uri,"PATCH", $data)
   
}
$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition
write-verbose "Running $folder\TcmExecWithLogin.ps1"
& "$folder\TcmExecWithLogin.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -LoginCreds "$TestUserUid,$TestUserPwd" -SettingsName $SettingsName -BuildNumber $BuildNumber -BuildDefinition $BuildDefinition
write-verbose "Got the exit code from the TCM run of $LASTEXITCODE"
$url = "$Collection/$Teamproject"
$jsondata = Get-BuildDetailsByNumber -tfsUri $url -buildNumber $BuildNumber -username $TestUserUid -password $TestUserPwd
$buildId = $jsondata.id
write-verbose "The build ID is $buildId"
$newquality = "Test Passed"
if ($LASTEXITCODE -gt 0 )
{
    $newquality = "Test Failed"
}
 
write-verbose "The build quality is $newquality"
Set-BuildQuality -tfsUri $url  -buildID $buildId -quality $newquality -username $TestUserUid -password $TestUserPwd

Note: TcmExecWithLogin.ps1 is the same as in the In my last post

Summary

So with these changes the process is now

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Build ends
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
    3. When the test complete we set the TFS build quality

This means we can associate both unit and integration tests with a build and target our release at any stage in the pipeline, it pausing at the points manual approval is required without blocking the initiating build.

Stray white space in a ‘path to custom test adaptors’ will cause tests to fail on VSO vNext build

If you are providing a path to a custom test adaptor such as nUnit or Chutzpah for a TFS/VSO vNext build e.g. $(Build.SourcesDirectory)\packages, make sure you have no leading whitespace in the data entry form.

image

 

If you do have a space you will see an error log like this as the adaptor cannot be found as the command line generated is malformed

2015-07-13T16:11:32.8986514Z Executing the powershell script: C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\tasks\VSTest\1.0.16\VSTest.ps1
2015-07-13T16:11:33.0727047Z ##[debug]Calling Invoke-VSTest for all test assemblies
2015-07-13T16:11:33.0756512Z Working folder: C:\a\0549426d
2015-07-13T16:11:33.0777083Z Executing C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "C:\a\0549426d\UnitTestDemo\WebApp.Tests\Scripts\mycode.tests.js"  /TestAdapterPath: C:\a\0549426d\UnitTestDemo\Chutzpah /logger:trx
2015-07-13T16:11:34.3495987Z Microsoft (R) Test Execution Command Line Tool Version 12.0.30723.0
2015-07-13T16:11:34.3505995Z Copyright (c) Microsoft Corporation.  All rights reserved.
2015-07-13T16:11:34.3896000Z ##[error]Error: The /TestAdapterPath parameter requires a value, which is path of a location containing custom test adapters. Example:  /TestAdapterPath:c:\MyCustomAdapters
2015-07-13T16:11:36.5808275Z ##[error]Error: The test source file "C:\a\0549426d\UnitTestDemo\Chutzpah" provided was not found.
2015-07-13T16:11:37.0004574Z ##[error]VSTest Test Run failed with exit code: 1
2015-07-13T16:11:37.0094570Z ##[warning]No results found to publish.

    Fix for 500 internal errors when trying to trigger a Release Management pipeline from a build via the REST API

    With the help of the Release Management team at Microsoft I now have a working REST based automated TFS Build to Release Management pipeline. Previously we were using a TFS automated build and then manually triggering our agent based Release Management pipeline. When we moved to a vNext PS/DSC based RM pipeline I took the chance to automate the link using REST via a PowerShell script to trigger the initial deployment. However, I hit problem, first with a stupid 401 permission error and later with a much stranger 500 internal server error.

    Fixing the 401 error

    The first problem was that in the InitiateReleaseFromBuild.ps1 script defaults to a hardcoded username and password. You should really be using the current credentials. To do this make sure the lines around line60 in the script are as shown below (or enter valid credentials if you don’t want to use default credentials)

    $wc = New-Object System.Net.WebClient
    $wc.UseDefaultCredentials = $true
    # rmuser should be part rm users list and he should have permission to trigger the release.
    #$wc.Credentials = new-object System.Net.NetworkCredential("rmuser", "rmuserpassword", "rmuserdomain")

    Fixing the 500 error

    The 500 error was stranger. Turns out the issue was the registration of our TFS server in Release Management.

    Using the dialogs in the RM client we has registered our TFS server, this had generated the URL https://tfs.domain.com:443/tfs. If we ran the InitiateReleaseFromBuild.ps1 script with this URL set as a parameter we got the 500 error, the RM logs showed the workflow could not start. Eventually we realised it was because RM thought it could not access the TFS server. So the problem was that at some point  between the script being run and the RM server processing the URL the :443 had been removed; presumably because this is the default for HTTPS and some layer was being ‘helpful’. This meant that the RM server was trying to string match the URL https://tfs.domain.com/tfs against https://tfs.domain.com:443/tfs which failed, hence the workflow failed.

    The fix was to edit the TFS registration in RM to remove the port number, leave the field empty (not that obvious as the dialog completes this field for you when you select HTTPS)

    image

    Once this was done the URL matching worked and the release pipeline triggered as expected.

    Strange TFS build process template editing issue with Typemock

    Had a strange issue today while editing our standard TFS 2013 XAML build process template to add an optional post drop script block to allow a Release Management pipeline to be triggered via REST. Our standard template includes a block for enabling and disabling Typemock, after editing our template to add the new script block (nowhere near the Typemock section) our builds failed with the error

    TF215097: An error occurred while initializing a build for build definition \BM\ISS.Expenses.Main.CI: Exception Message: Cannot set unknown member 'TypeMock.TFS2013.TypeMockStart.DisableAutoLink'. (type XamlObjectWriterException) Exception Stack Trace: at System.Xaml.XamlObjectWriter.WriteStartMember(XamlMember property) 

    It took ages to find the issue, we hunted for badly formed XAML, but the issue turned out to be that when ever we opened the template in Visual Studio 2013 it added the highlighted property

     

    <If Condition="[UseTypemock = True]" DisplayName="If using Typemock" sap2010:WorkflowViewState.IdRef="If_8">
      <If.Then>
       <Sequence DisplayName="Enabling Typemock" sap2010:WorkflowViewState.IdRef="Sequence_16">
          <tt:TypeMockRegister AutoDeployDir="[TypemockAutoDeployDir]" Company="[TypemockCompany]" sap2010:WorkflowViewState.IdRef="TypeMockRegister_1" License="[TypemockLicense]" />
          <tt:TypeMockStart DisableAutoLink="{x:Null}" EvaluationFolder="{x:Null}" Link="{x:Null}" LogLevel="{x:Null}" LogPath="{x:Null}" ProfilerLaunchedFirst="{x:Null}" Target="{x:Null}" Verbosity="{x:Null}" Version="{x:Null}" AutoDeployDir="[TypemockAutoDeployDir]" sap2010:WorkflowViewState.IdRef="TypeMockStart_1" />
         </Sequence>
      </If.Then>
    </If>

    It should have been

    <If Condition="[UseTypemock = True]" DisplayName="If using Typemock" sap2010:WorkflowViewState.IdRef="If_8">
      <If.Then>
        <Sequence DisplayName="Enabling Typemock" sap2010:WorkflowViewState.IdRef="Sequence_16">
           <tt:TypeMockRegister AutoDeployDir="[TypemockAutoDeployDir]" Company="[TypemockCompany]" sap2010:WorkflowViewState.IdRef="TypeMockRegister_1" License="[TypemockLicense]" />
           <tt:TypeMockStart EvaluationFolder="{x:Null}" Link="{x:Null}" LogLevel="{x:Null}" LogPath="{x:Null}" ProfilerLaunchedFirst="{x:Null}" Target="{x:Null}" Verbosity="{x:Null}" Version="{x:Null}" AutoDeployDir="[TypemockAutoDeployDir]" sap2010:WorkflowViewState.IdRef="TypeMockStart_1" />
        </Sequence>
      </If.Then>
    </If>

    All I can assume is that this is due to some assembly mismatch between the Typemock DLLs linked to the XAML build process template and those on my development PC.

    The fix for now is to do the editing in a text editor, or at least checking the file to make sure the property has not been edited before it is checked in.

    MSDeploy Parameters.xml can only replace web.config values is a value is already set

    If you are using a parameters.xml file to set value with MSDeploy I have just found a gotcha. You need some value in the web.config file, not just an empty XML tag, else the replacement fails. So to explain…

    I had the following parameters.xml file, and use Release Management to replace the __TAG__ values at deployment time.

    <parameters>
      <parameter name="Domain" description="Please enter the name of the domain" defaultvalue="__Domain__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/Web.Properties.Settings/setting[@name='Domain']/value/text()" />
      </parameter>

      <parameter name="AdminGroups" description="Please enter the name of the admin group" defaultvalue="__AdminGroups__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/Web.Properties.Settings/setting[@name='AdminGroups']/value/text()" />
      </parameter>
    </parameters>

    If my web.config file (in the MSDeploy package to be transformed) was set to

    <applicationSettings>
        <Web.Properties.Settings>
          <setting name="Domain" serializeAs="String">
            <value>Blackmarble</value>
          </setting>
          <setting name="AdminGroups" serializeAs="String">
            <value />
          </setting>
        </BlackMarble.ISS.Expenses.Web.Properties.Settings>
      </applicationSettings>

    or

    <applicationSettings>
        <Web.Properties.Settings>
          <setting name="Domain" serializeAs="String">
            <value>Blackmarble</value>
          </setting>
          <setting name="AdminGroups" serializeAs="String">
            <value></value>
          </setting>
        </BlackMarble.ISS.Expenses.Web.Properties.Settings>
      </applicationSettings>

    only the Domain setting was set.

    To get both set I had to have a value for each property, even though they were being reset at deployment.

    <applicationSettings>
        <Web.Properties.Settings>
          <setting name="Domain" serializeAs="String">
            <value>DummyDomain</value>
          </setting>
          <setting name="AdminGroups" serializeAs="String">
            <value>DummyAdmins</value>
          </setting>
        </BlackMarble.ISS.Expenses.Web.Properties.Settings>
      </applicationSettings>

    Never seen that one before.

    A Visual Studio Extension to create MSDeploy parameters.xml files

    When you using MSdeploy you should create a parameters.xml file that exposes your web.config settings at the time of installation. This enables good deployment habits, build the product one and then set system specific values using deployment tools. The problem is that this parameters.xml file is a pain to write, it is a series of XML blocks that contain XPath to find the entries to replace, typo’s are easy to introduce.

    A ripe candidate for automation, but I could not find a tool to do it, so I wrote one for Visual Studio 2013 and 2015. You can find the source on GitHub andthe actual VSIX package in the Visual Studio Gallery.

    So what does it do?

    Once it is installed, if you right click on a web.config file you will see a context menu option to generate a parameters.xml file, click it, if the file does not exist it will be generated and added to the current project. Entries will be made for all appSettings and any custom applicationSettings blocks found in the web.config. The actual web.config values will be replaced with __TAGS__ to be set via Release Management or your tool of choice.

    So the web.config file

    <configuration>
      <applicationSettings>
        <Service.Properties.Settings>
          <setting name="Directory1" serializeAs="String">
            <value>C:\ABC1111</value>
          </setting>
          <setting name="Directory2" serializeAs="String">
            <value>C:\abc2222</value>
          </setting>
        </Service.Properties.Settings>
      </applicationSettings>
      <appSettings>
        <add key="APPSETTING1" value="123" />
        <add key="AppSetting2" value="456" />
      </appSettings>
    </configuration>  

    it generates the parameters.xml

    <parameters>
      <parameter name="APPSETTING1" description="Description for APPSETTING1" defaultvalue="__APPSETTING1__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/appSettings/add[@key='APPSETTING1']/@value" />
      </parameter>

      <parameter name="AppSetting2" description="Description for AppSetting2" defaultvalue="__APPSETTING2__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/appSettings/add[@key='AppSetting2']/@value" />
      </parameter>

      <parameter name="Directory1" description="Description for Directory1" defaultvalue="__DIRECTORY1__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory1']/value/text()" />
      </parameter>

      <parameter name="Directory2" description="Description for Directory2" defaultvalue="__DIRECTORY2__" tags="">
        <parameterentry kind="XmlFile" scope="\\web.config$" match="/configuration/applicationSettings/Service.Properties.Settings/setting[@name='Directory2']/value/text()" />
      </parameter>

    </parameters>

    If a parameters.xml file already exists then you are prompted first if you wish to replace it, if you say no, then you  are prompted if you wish to add any new entries in the web.config, or do nothing.

    All the work is done via an XSL Transform, so if you need to transform extra settings just add to the embedded XSLT resource and rebuild the VSIX package.

    So the tool won’t do everything, but should get you close to the file you need.

    Running StyleCop from the command line and in a TFS 2015 vNext build

    Virtually any automated build will require some customisation beyond a basic compile. So as part of my upcoming Techorama session on TFS 2015 vNext build I need a demo of using a custom script as part of the build process. Two common customisations we use are version stamping of assemblies and running code analysis tools. For vNext build there is already a sample of version stamping, so I thought getting StyleCop running would be a good sample.

    The problem

    Customisation in vNext build is based around running a script, in the case of a Windows based build agents this a PowerShell script. The problem with StyleCop is that it does not provide a command line iterface. The  StyleCop CodePlex project provides only a Visual Studio add-in. There is also the ALM Ranger’s TFS community custom build  activity, but I could find no current command line interface projects.

    So I needed to build one.

    Step 1 – Create a command line

    So my first step was to create a command line version of StyleCop. I chose to use the community build activity as a starting point. I had planned to do this all in PowerShell, but quickly found that the conversion of parameter object types and the handling of the events StyleCop uses was a bit messy. So I decided to write a wrapper class in C# that presented the same parameters as the old TFS build activity, basically take the old code and remove the Windows Workflow logic. I then provided a Main (args) method to expose the object to the command line such that it was easy to provide the required parameters.

    This can all be found on my GitHub site.

    Note on solution structure: As I wanted this to work for PowerShell and the command prompt I had to place the Main(args[]) method .EXE entry point in a project that built an EXE and all the rest of the wrapper code in one that built a .DLL. This is because you cannot load a type in PowerShell using add-type from an assembly built as an EXE, you get a EXTENSION_NOT_SUPPORTED exception. It means there are two projects (a DLL and an EXE) when I would really have like a single one (the EXE)

    So I now had a command line I could call from my PowerShell script

    StyleCopCmdLine --f="File1.cs" "File2.cs" --s="AllSettingsEnabled.StyleCop"

    A good starting point. However,  more a TFS build it makes more sense to call StyleCop directly in the PowerShell, why shell out to a command prompt to run an EXE when your can run the code directly in PowerShell?

    Step 2 – Create a simple PowerShell script

    The PowerShell required to run StyleCop using the wrapper is simple, just providing the same parameters as used for the EXE.

    Add-Type -Path "StyleCopWrapper.dll"

     

    $scanner = new-object StyleCopWrapper.Wrapper
    $scanner.MaximumViolationCount = 1000
    $scanner.ShowOutput = $true
    $scanner.CacheResults = $false
    $scanner.ForceFullAnalysis = $true
    $scanner.XmlOutputFile = "$pwd\out.xml"
    $scanner.LogFile = "$pwd\log.txt"
    $scanner.SourceFiles =  @("file1.cs", "file2.cs") )
    $scanner.SettingsFile = "settings.stylecop"
    $scanner.AdditionalAddInPaths = @("C:\Program Files (x86)\StyleCop 4.7" )
    $scanner.TreatViolationsErrorsAsWarnings = $false

    $scanner.Scan()

    write-host ("Succeeded [{0}]" -f $scanner.Succeeded)
    write-host ("Violation count [{0}]" -f $scanner.ViolationCount)

    See the GitHub site’s WIKI for the usage details.

    Step 3 – Create a vNext build PowerShell script

    So now we have the basic tools we need to run StyleCop from a TFS vNext build, but we do need a more complex script.

    The script you use is up to you, mine looks for .csproj files and runs StyleCop recursively from the directories containing the .csproj files. This means I can have a different  setting.stylecop file for each project. In general I have more strict rules on production code than unit test e.g. for unit tests I am not bother about the XML method documentation, but for production code I make sure they are present and match the method parameters.

    Note: As the script just uses parameters and environment variable it is easy to test outside TFS build, a great improvement over the old build system

    #
    # Script to allow StyleCop to be run as part of the TFS vNext build
    #
    [CmdletBinding()]
    param
    (
        # We have to pass this boolean flag as string, we cast it before we use it
        # have to use 0 or 1, true or false
        [string]$TreatStyleCopViolationsErrorsAsWarnings = 'False'
    )

    # local test values, should be commented out in production
    #$Env:BUILD_STAGINGDIRECTORY = "C:\drops"
    #$Env:BUILD_SOURCESDIRECTORY = "C:\code\MySolution"

    if(-not ($Env:BUILD_SOURCESDIRECTORY -and $Env:BUILD_STAGINGDIRECTORY))
    {
        Write-Error "You must set the following environment variables"
        Write-Error "to test this script interactively."
        Write-Host '$Env:BUILD_SOURCESDIRECTORY - For example, enter something like:'
        Write-Host '$Env:BUILD_SOURCESDIRECTORY = "C:\code\MySolution"'
        Write-Host '$Env:BUILD_STAGINGDIRECTORY - For example, enter something like:'
        Write-Host '$Env:BUILD_STAGINGDIRECTORY = "C:\drops"'
        exit 1
    }

    # pickup the build locations from the environment
    $stagingfolder = $Env:BUILD_STAGINGDIRECTORY
    $sourcefolder = $Env:BUILD_SOURCESDIRECTORY

    # have to convert the string flag to a boolean
    $treatViolationsErrorsAsWarnings = [System.Convert]::ToBoolean($TreatStyleCopViolationsErrorsAsWarnings)

    Write-Host ("Source folder (`$Env)  [{0}]" -f $sourcefolder) -ForegroundColor Green
    Write-Host ("Staging folder (`$Env) [{0}]" -f $stagingfolder) -ForegroundColor Green
    Write-Host ("Treat violations as warnings (Param) [{0}]" -f $treatViolationsErrorsAsWarnings) -ForegroundColor Green
     
    # the overall results across all sub scans
    $overallSuccess = $true
    $projectsScanned = 0
    $totalViolations = 0

    # load the StyleCop classes, this assumes that the StyleCop.DLL, StyleCop.Csharp.DLL, # StyleCop.Csharp.rules.DLL in the same folder as the StyleCopWrapper.dll $folder = Split-Path -parent $MyInvocation.MyCommand.Definition Write-Host ("Loading from folder from [{0}]" -f $folder) -ForegroundColor Green $dllPath = [System.IO.Path]::Combine($folder,"StyleCopWrapper.dll") Write-Host ("Loading DDLs from [{0}]" -f $dllPath) -ForegroundColor Green Add-Type -Path $dllPath

    # Set the common scan options,
    $scanner.MaximumViolationCount = 1000
    $scanner.ShowOutput = $true
    $scanner.CacheResults = $false
    $scanner.ForceFullAnalysis = $true
    $scanner.AdditionalAddInPaths = @($pwd) # in in local path as we place stylecop.csharp.rules.dll here
    $scanner.TreatViolationsErrorsAsWarnings = $treatViolationsErrorsAsWarnings

    # look for .csproj files
    foreach ($projfile in Get-ChildItem $sourcefolder -Filter *.csproj -Recurse)
    {
       write-host ("Processing the folder [{0}]" -f $projfile.Directory)

       # find a set of rules closest to the .csproj file
       $settings = Join-Path -path $projfile.Directory -childpath "settings.stylecop"
       if (Test-Path $settings)
       {
            write-host "Using found settings.stylecop file same folder as .csproj file"
            $scanner.SettingsFile = $settings
       }  else
       {
           $settings = Join-Path -path $sourcefolder -childpath "settings.stylecop"
           if (Test-Path $settings)
           {
                write-host "Using settings.stylecop file in solution folder"
                $scanner.SettingsFile = $settings
           } else
           {
                write-host "Cannot find a local settings.stylecop file, using default rules"
                $scanner.SettingsFile = "." # we have to pass something as this is a required param
           }
       }

       $scanner.SourceFiles =  @($projfile.Directory)
       $scanner.XmlOutputFile = (join-path $stagingfolder $projfile.BaseName) +".stylecop.xml"
       $scanner.LogFile =  (join-path $stagingfolder $projfile.BaseName) +".stylecop.log"
       
       # Do the scan
       $scanner.Scan()

        # Display the results
        Write-Host ("`n")
        write-host ("Base folder`t[{0}]" -f $projfile.Directory) -ForegroundColor Green
        write-host ("Settings `t[{0}]" -f $scanner.SettingsFile) -ForegroundColor Green
        write-host ("Succeeded `t[{0}]" -f $scanner.Succeeded) -ForegroundColor Green
        write-host ("Violations `t[{0}]" -f $scanner.ViolationCount) -ForegroundColor Green
        Write-Host ("Log file `t[{0}]" -f $scanner.LogFile) -ForegroundColor Green
        Write-Host ("XML results`t[{0}]" -f $scanner.XmlOutputFile) -ForegroundColor Green

        $totalViolations += $scanner.ViolationCount
        $projectsScanned ++
       
        if ($scanner.Succeeded -eq $false)
        {
          # any failure fails the whole run
          $overallSuccess = $false
        }

    }

    # the output summary
    Write-Host ("`n")
    if ($overallSuccess -eq $false)
    {
       Write-Error ("StyleCop found [{0}] violations across [{1}] projects" -f $totalViolations, $projectsScanned)
    }
    elseif ($totalViolations -gt 0 -and $treatViolationsErrorsAsWarnings -eq $true)
    {
        Write-Warning ("StyleCop found [{0}] violations warnings across [{1}] projects" -f $totalViolations, $projectsScanned)
    }
    else
    {
       Write-Host ("StyleCop found [{0}] violations warnings across [{1}] projects" -f $totalViolations, $projectsScanned) -ForegroundColor Green
    }

    Step 4 – Adding a the script to the repo

    To use the script it needs (and any associated files) to be placed in your source control. In my case it meant I create a folder called StyleCop off the root of my TFS 2015 CTP’s Git repo and in it placed the following files

    • PowerShell.ps1 – my script file
    • StyleCop.dll – the main StyleCop assembly taken from c:\program files (x86)\StyleCop 4.7. By placing it here it means we don’t need to actually install StyleCop on the build machine
    • StyleCop.csharp.dll – also from c:\program files (x86)\StyleCop 4.7
    • StyleCop.csharp.rules.dll – also from c:\program files (x86)\StyleCop 4.7
    • StyleCopWrapper.dll – the wrapper assembly from my GitHub site

    Step 5 – Adding the script to a build process

    Once the script is in the repo adding a new step to a vNext build is easy.

    • In a browser select the Build.vNext menu options
    • The build explorer will be show, right click on the build you wish to add a step to and select edit
    • Press the ‘Add build step’ button. The list of steps will be show, pick PowerShell

      image
    • As the script is in the repo we can reference it in the new step. in my case I set the script file name to

                            StyleCop/PowerShell.ps1
    • My script takes one parameter, if we should treat StleCop violations as warnings, this is set as the script argument. Note I am using a build variable $(ViolationsAsWarnings) set to a string value ‘True’ or ‘False’, so I have one setting for the whole build script. Though a boolean parameter would be nice it seems I can only pass in strings as build variables, so I do the conversion to a boolean inside the script.

                            -TreatStyleCopViolationsErrorsAsWarnings $(ViolationsAsWarnings)

      image

    Step 6 - Running the build

    My test solution has two projects, with different settings.stylecop files. Once the new step was added to my build I could queue a build, by altering $(ViolationsAsWarnings)  variable I could make the build pass for fail.

            

    image

           image

    The detailed StyleCop result are available in the build log and are also placed in the drops folder in an XML format.

    Note: One strange behaviour is that when you test the script outside TFS build you get a .XML and .LOG file for each project scanned. In TFS build you only see the .XML file in the drops folder, this I think is because the .LOG has been redirected into the main TFS vNext build logs.

    Summary

    So now I have a way to run StyleCop within a TFS vNext build.

    Using these techniques there are no end of tools that can be wired into the build process, and I must say it is far easier than the TFS 2010, 2012, 2013 style workflow customisation.