But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Running CodeUI tests on a VM with on remote desktop session open as part of a vNext build

If you want to run CodeUI tests as part of a build you need to make sure the device running the test has access to the UI, for remote VMs this means having a logged in session open and the build/test agent running interactivally. Problem is what happens when you disconnect the session. UNless you manage it you will get the error

Automation engine is unable to playback the test because it is not able to interact with the desktop. This could happen if the computer running the test is locked or it’s remote session window is minimized

In the past I would use a standard TFS Lab Management Environment to manage this,you just check a box to say the VM/PC is running coded UI tests and it sorts out the rest. However, with the advent of vNext build and the move away from Lab Manager this seems overly complex.

It is not a perfect solution but this works

  1. Make sure the VM autologs in and starts your build/test agents in interactive mode (I used SysInternal AutoLogin to set this up)
  2. I connect to the session and make sure all is OK, but I then disconnect redirecting the session
    • To get my session ID, at the command prompt, I use the command query user
    • I then redirect the session tscon.exe RDP-Tcp#99 /dest:console, where RDP-Tcp#99 is my session ID
  3. Once I was disconnected my CodeUI test still run

I am sure I can get a slicker way to do this, but it does fix the immediate issue

Updated:

This bit of Powershell code could be put in a shortcut on the desktop to do the job, you will want to run the script as administrator

$OutputVariable = (query user) | Out-String

$session = $OutputVariable.Substring($OutputVariable.IndexOf("rdp-tcp#")).Split(" ")[0]

& tscon.exe $session /dest:console

Chrome extension to help with exploratory testing

One of the many interesting announcements at Connect() today was that the new Microsoft Chrome Extension for Exploratory Testing  is  available in the Chrome Store

This is a great tool if you use VSO, sorry VSTS, allowing an easy way to ‘kick the tyres’ on your application, logging any bugs directly back to VSTS as Bug work items.

image

Best of all, it makes it easy to test your application on other platforms with the link to Perfecto Mobile. Just press the device button, login and you can launch a session on a real physical mobile device to continue your exploratory testing.

image

Only down side I can see that that, if like me, you would love this functionality for on-premises TFS we need to wait a while, this first preview only support VSTS.

Release Manager - New deployment is not allowed as an another deployment is in progress

Whilst working with a vNext Release Management pipeline I started seeing the error

Microsoft.TeamFoundation.Release.Common.Helpers.OperationFailedException:
New deployment is not allowed as an another deployment is in progress.
Retry the deployment after sometime.

Problem was I could not see any blocked or paused deployment releases. All Internet searches mentioned multiple pipelines that share components, but this was not the issue.

Eventually I found the issue, my release pipeline included a step that ran CodedUI tests via TCM, hence a previous running of this template had triggered the test via TCM, but they had stalled. I found this by looking in MTM.

image

Release Management was just saying the release was rejected with the above error message, no clue about the unfinished test run. Not that helpful.

You might have expect Release Management to return only after the test had timed out, but that is only down to whether you set the release pipeline to wait or not, I had set mine not to wait.

Once I stopped this test run via MTM all was OK.

Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build

This article was first published on the Microsoft’s UK Developers site as Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build

With the advent of vNext build in TFS 2015 and Visual Studio Online running unit tests that are not MSTest based within your build process is far more straightforward than it used to be. No longer do you have to use custom XAML build activities or tell all your TFS build controllers where the test runner assemblies are. The ‘out the box’ vNext build Visual Studio Test task will automatically load any test adaptors it finds in the path specified for test runners in its advanced properties, a path that can be populated via NuGet.

Running nUnit tests

All this means that to find and run MSTest and nUnit tests as part of your build all you have to do is as follows

  1. Create a solution that contains a project with MStest and nUnit tests, in my sample this is a MVC web application project with its automatically created MSTest unit tests project.
  2. In the test project add some nUnit tests. Use NuGet to add the references to nUnit to the test project so it compiles.
  3. Historically in your local Visual Studio instance you needed to install the nUnit Test Runner VSIX package from Visual Studio Gallery – this allows Visual Studio to discover your nUnit tests, as well as any MSTest ones, and run them via the built in Test Explorer

    image

    IMPORTANT Change –
    However installing this VSIX package is no longer required. If you use Nuget to add the nUnit Test Runner to the solution, as well as the nUnit package itself, then Visual Studio can find the nUnit tests without the VSIX package. This is useful but not world changing on your development PC, but when on the build box it means the NuGet restore will make sure the nUnit test adapter assemblies are pulled down onto the local build boxes file system and used to find tests with no extra work.

    Note
    : If you still want to install the VSIX package on your local Visual Studio instance you can, it is just you don’t have to.
  4. Check in your solution into TFS/VSO source control. It does not matter if it is TFVC or Git based
  5. Create a new vNext build using the Visual Studio template
  6. You can leave most of the parameters on default setting. But you do need to edit the Visual Studio Test task’s advanced settings to point at the NuGet packages folder for your solution (which will be populated via NuGet restore) so the custom nUnit test adaptor can be found i.e. usually setting it to  $(Build.SourcesDirectory)\packages

    image
  7. The build should run and find your tests, the MStest ones because they are built in and the nUnit ones because it found the custom test adaptor due to the NuGet restore being done prior to the build. The test results can be found on the build summary page

    image

 

But what if you want run Jasmine.JS test?

If you want to run Jasmine JavaScript unit tests the process is basically the same. The only major difference is that you do still need to install the Chutzpah Test runner on your local Visual Studio as a VSIX package to run the tests locally. There is a NuGet package for the Chutzpah test runner so you can avoid having to manually unpack the VSIX and get it into source control to deploy it to the build host (unless you really want to follow this process), but this package does not currently enable Visual Studio to find the Jasmine tests without the VSIX extension being installed, or at least it didn’t for me.

Using the solution I used before

  1. Use NuGet to add Jasmine.JS to the test project
  2. Add a test file to the test project e.g. mycode.tests.js (adding any JavaScript references needed to find any script code under test in the main WebApp project)
  3. Install the Chutzpah Test runner in your local Visual Studio as a VSIX extension, restart Visual Studio
  4. You should now be able to see and run the Jasmine test run in the test runner as well as the MSTest and nUnit tests.

    image
  5. Add the NuGet package for the Chutzpah test runner to your solution, this is a solution level package, so does not need to be associated with any project.
  6. Check the revised code into source control
  7. In your vNext build add another Visual Studio Test task, set the test assembly to match your javascript test naming convention e.g. **\*.tests.js and the path to the custom test adaptor to $(Build.SourcesDirectory)\packages (as before)

    image
  8. Run the revised build.

    image
  9. You should see the two test tasks run and a pair of test results in the summary for the build.

So now hopefully you should find this a more straight forward way to added testing to your vNext builds. Allowing easy use of both your own build boxes and the hosted build service for VSO with testing frameworks they do not support ‘out the box’

An alternative to setting a build quality on a TFS vNext build

TFS vNext builds do not have a concept of build quality unlike the old XAML based builds. This is an issue for us as we used the changing of the build quality as signal to test a build, or to mark it as released to a client (this was all managed with my TFS Alerts DSL to make sure suitable emails and build retention were used).

So how to get around this problem with vNext?

I have used Tag on builds, set using the same REST API style calls as detailed in my post on Release Management vNext templates. I also use the REST API to set the retention on the build, so I actually now don’t need to manage this via the alerts DSL.

The following script, if used to wrapper the calling of integration tests via TCM, should set the tags and retention on a build


function Get-BuildDetailsByNumber
{
    param
    (
        $tfsUri ,
        $buildNumber,
        $username,
        $password

    )

    $uri = "$($tfsUri)/_apis/build/builds?api-version=2.0&buildnumber=$buildNumber"

    $wc = New-Object System.Net.WebClient
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
    write-verbose "Getting ID of $buildNumber from $tfsUri "

    $jsondata = $wc.DownloadString($uri) | ConvertFrom-Json
    $jsondata.value[0]
 
}

function Set-BuildTag
{
    param
    (
        $tfsUri ,
        $buildID,
        $tag,
        $username,
        $password

    )

 
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
   
    write-verbose "Setting BuildID $buildID with Tag $tag via $tfsUri "

    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)/tags/$($tag)?api-version=2.0"

    $data = @{value = $tag } | ConvertTo-Json

    $wc.UploadString($uri,"PUT", $data)
   
}

function Set-BuildRetension
{
    param
    (
        $tfsUri ,
        $buildID,
        $keepForever,
        $username,
        $password

    )

 
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    if ($username -eq $null)
    {
        $wc.UseDefaultCredentials = $true
    } else
    {
        $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
    }
   
    write-verbose "Setting BuildID $buildID with retension set to $keepForever via $tfsUri "

    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)?api-version=2.0"
    $data = @{keepForever = $keepForever} | ConvertTo-Json
    $response = $wc.UploadString($uri,"PATCH", $data)
   
}


# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose

$ErrorActionPreference = 'Continue' # this controls if any test failure cause the script to stop

 

$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition

write-verbose "Running $folder\TcmExec.ps1"

 

& "$folder\TcmExec.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -SettingsName $SettingsName write-verbose "TCM exited with code '$LASTEXITCODE'"
$newquality = "Test Passed"
$tag = "Deployed to Lab"
$keep = $true
if ($LASTEXITCODE -gt 0 )
{
    $newquality = "Test Failed"
    $tag = "Lab Deployed failed"
    $keep = $false
}
write-verbose "Setting build tag to '$tag' for build $BuildNumber"


$url = "$Collection/$Teamproject"
$jsondata = Get-BuildDetailsByNumber -tfsUri $url -buildNumber $BuildNumber #-username $TestUserUid -password $TestUserPwd
$buildId = $jsondata.id
write-verbose "The build $BuildNumber has ID of $buildId"
 
write-verbose "The build tag set to '$tag' and retention set to '$key'"
Set-BuildTag -tfsUri $url  -buildID $buildId -tag $tag #-username $TestUserUid -password $TestUserPwd
Set-BuildRetension -tfsUri $url  -buildID $buildId  -keepForever $keep #-username $TestUserUid -password $TestUserPwd

# now fail the stage after we have sorted the logging
if ($LASTEXITCODE -gt 0 )
{
    Write-error "Test have failed"
}

If all the tests pass we see the Tag being added and the retention being set, if they fail just a tag should be set

image

$ErrorActionPreference = 'Continue'

Running Microsoft Test Manager Test Suites as part of a vNext Release pipeline - Part 2

In my last post I discussed how you could wire TCM tests into a Release Management vNext pipeline. The problem with the script I provided, as I noted, was that the deployment was triggered synchronously by the build i.e. the build/release process was:

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Wait while the RM pipeline completed
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
  3. When RM completed the TFS build completes

This process raised a couple of problems

  • You cannot associate the integration tests with the build as TCM only allow association with completed successful builds. When TCM finishes in this model the build is still in progress.
  • You have to target only the first automated stage of the pipeline, else the build will be held as ‘in progress’ until all the release stages have complete, which may be days if there are manual approvals involved

The script InitiateReleaseFromBuild

These problems can all be fixed by altering the PowerShell that triggers the RM pipeline so that it does not wait for the deployment to complete, so the TFS build completes as soon as possible.

This is done by passing in an extra parameter which is set in TFS build

param(
    [string]$rmserver = $Args[0],
    [string]$port = $Args[1], 
    [string]$teamProject = $Args[2],  
    [string]$targetStageName = $Args[3],
    [string]$waitForCompletion = $Args[4]
)

cls
$teamFoundationServerUrl = $env:TF_BUILD_COLLECTIONURI
$buildDefinition = $env:TF_BUILD_BUILDDEFINITIONNAME
$buildNumber = $env:TF_BUILD_BUILDNUMBER


"Executing with the following parameters:`n"
"  RMserver Name: $rmserver"
"  Port number: $port"
"  Team Foundation Server URL: $teamFoundationServerUrl"
"  Team Project: $teamProject"
"  Build Definition: $buildDefinition"
"  Build Number: $buildNumber"
"  Target Stage Name: $targetStageName`n"
"  Wait for RM completion: $waitForCompletion`n"

$wait = [System.Convert]::ToBoolean($waitForCompletion)
$exitCode = 0

trap
{
  $e = $error[0].Exception
  $e.Message
  $e.StackTrace
  if ($exitCode -eq 0) { $exitCode = 1 }
}

$scriptName = $MyInvocation.MyCommand.Name
$scriptPath = Split-Path -Parent (Get-Variable MyInvocation -Scope Script).Value.MyCommand.Path

Push-Location $scriptPath   

$server = [System.Uri]::EscapeDataString($teamFoundationServerUrl)
$project = [System.Uri]::EscapeDataString($teamProject)
$definition = [System.Uri]::EscapeDataString($buildDefinition)
$build = [System.Uri]::EscapeDataString($buildNumber)
$targetStage = [System.Uri]::EscapeDataString($targetStageName)

$serverName = $rmserver + ":" + $port
$orchestratorService = "http://$serverName/account/releaseManagementService/_apis/releaseManagement/OrchestratorService"

$status = @{
    "2" = "InProgress";
    "3" = "Released";
    "4" = "Stopped";
    "5" = "Rejected";
    "6" = "Abandoned";
}

$uri = "$orchestratorService/InitiateReleaseFromBuild?teamFoundationServerUrl=$server&teamProject=$project&buildDefinition=$definition&buildNumber=$build&targetStageName=$targetStage"
"Executing the following API call:`n`n$uri"

$wc = New-Object System.Net.WebClient
$wc.UseDefaultCredentials = $true
# rmuser should be part rm users list and he should have permission to trigger the release.

#$wc.Credentials = new-object System.Net.NetworkCredential("rmuser", "rmuserpassword", "rmuserdomain")

try
{
    $releaseId = $wc.DownloadString($uri)

    $url = "$orchestratorService/ReleaseStatus?releaseId=$releaseId"

    $releaseStatus = $wc.DownloadString($url)


    if ($wait -eq $true)
    {
        Write-Host -NoNewline "`nReleasing ..."

        while($status[$releaseStatus] -eq "InProgress")
        {
            Start-Sleep -s 5
            $releaseStatus = $wc.DownloadString($url)
            Write-Host -NoNewline "."
        }

        " done.`n`nRelease completed with {0} status." -f $status[$releaseStatus]
    } else {

        Write-Host -NoNewline "`nTriggering Release and exiting"
    }

}
catch [System.Exception]
{
    if ($exitCode -eq 0) { $exitCode = 1 }
    Write-Host "`n$_`n" -ForegroundColor Red
}

if ($exitCode -eq 0)
{
    if ($wait -eq $true)
    {
        if ($releaseStatus -eq 3)
        {
          "`nThe script completed successfully. Product deployed without error`n"
        } else {
            Write-Host "`nThe script completed successfully. Product failed to deploy`n" -ForegroundColor Red
            $exitCode = -1 # reset the code to show the error
        }
    } else {
        "`nThe script completed successfully. Product deploying`n"
    }
}
else
{
  $err = "Exiting with error: " + $exitCode + "`n"
  Write-Host $err -ForegroundColor Red
}

Pop-Location

exit $exitCode

The Script TcmExecWrapper

A change is also required in the wrapper script I use to trigger the TCM test run. We need to check the exit code from the inner TCM PowerShell script and update the TFS build quality appropriately.

To this I use the new REST API in TFS 2015 as this is far easier than using the older .NET client API. No DLLs to distribute.

It is worth noticing that

  • I pass the credentials into the script from RM that are used to talk to the TFS server. This is because I am running my tests in a network isolated TFS Lab Environment, this means I am in the wrong domain to see the TFS server without providing login details. If you are not working cross domain you could just use Default Credentials.
  • RM only passes the BuildNumber into the script e.g. MyBuild_1.2.3.4, but the REST API need the build id to set the quality. Hence the need for function Get-BuildDetailsByNumber to get the id from the name
# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose
function Get-BuildDetailsByNumber
{
    param
    (
        $tfsUri ,
        $buildNumber,
        $username,
        $password
    )
    $uri = "$($tfsUri)/_apis/build/builds?api-version=2.0&buildnumber=$buildNumber"
    $wc = New-Object System.Net.WebClient
    #$wc.UseDefaultCredentials = $true
    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
   
    write-verbose "Getting ID of $buildNumber from $tfsUri "
    $jsondata = $wc.DownloadString($uri) | ConvertFrom-Json
    $jsondata.value[0]
 
}
function Set-BuildQuality
{
    param
    (
        $tfsUri ,
        $buildID,
        $quality,
        $username,
        $password
    )
    $uri = "$($tfsUri)/_apis/build/builds/$($buildID)?api-version=1.0"
    $data = @{quality = $quality} | ConvertTo-Json
    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = "application/json"
    #$wc.UseDefaultCredentials = $true
    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)
   
    write-verbose "Setting BuildID $buildID to quality $quality via $tfsUri "
    $wc.UploadString($uri,"PATCH", $data)
   
}
$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition
write-verbose "Running $folder\TcmExecWithLogin.ps1"
& "$folder\TcmExecWithLogin.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -LoginCreds "$TestUserUid,$TestUserPwd" -SettingsName $SettingsName -BuildNumber $BuildNumber -BuildDefinition $BuildDefinition
write-verbose "Got the exit code from the TCM run of $LASTEXITCODE"
$url = "$Collection/$Teamproject"
$jsondata = Get-BuildDetailsByNumber -tfsUri $url -buildNumber $BuildNumber -username $TestUserUid -password $TestUserPwd
$buildId = $jsondata.id
write-verbose "The build ID is $buildId"
$newquality = "Test Passed"
if ($LASTEXITCODE -gt 0 )
{
    $newquality = "Test Failed"
}
 
write-verbose "The build quality is $newquality"
Set-BuildQuality -tfsUri $url  -buildID $buildId -quality $newquality -username $TestUserUid -password $TestUserPwd

Note: TcmExecWithLogin.ps1 is the same as in the In my last post

Summary

So with these changes the process is now

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Build ends
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
    3. When the test complete we set the TFS build quality

This means we can associate both unit and integration tests with a build and target our release at any stage in the pipeline, it pausing at the points manual approval is required without blocking the initiating build.

Running Microsoft Test Manager Test Suites as part of a vNext Release pipeline

Also see Part 2 on how to address gotcha's in this process

When using Release Management there is a good chance you will want to run test suites as part of your automated deployment pipeline. If you are using a vNext PowerShell based pipeline you need a way to trigger the tests via PowerShell as there is no out the box agent to do the job.

Step 1 - Install a Test Agent

The first step is to make sure that the Visual Studio Test Agent is installed on the box you wish to run the test on. if you don’t already have a MTM Environment in place with a test agent then this can be done by creating a standard environment in Microsoft Test Manager. Remember you only need this environment to include the VM you want to run the test on, unless you want to also gather logs and events from our machines in the system. The complexity is up to you.

In my case I was using a network isolated environment so all this was already set up.

Step 2 - Setup the Test Suite

Once you have an environment you can setup your test suite and test plan in MTM to include the tests you wish to run. These can be unit test style integration tests or Coded UI it is up to you.

If you have a lot of unit tests to associate for automation remember the TCM.EXE command can make your life a lot easier

This post does not aim to be a tutorial on setting up test plans, have a look at the ALM Rangers guides for more details.

Step 3 -  The Release Management environment

This is where it gets a bit confusing, you have already set up a Lab Management environment, but you still need to setup the Release Management vNext environment. As I was using a network isolated Lab management environment this gets even more complex, but RM provides some tools to help

Again this is not a detailed tutorial. The key steps if you are using network isolation are

  1. Make sure that PowerShell on the VM is setup for remote access by running  winrm quickconfig
  2. In RM create a vNext environment
  3. Add each a new server, using it’s corporate LAN name from Lab Management with the PowerShell remote access port e.g. VSLM-1002-e7858e28-77cf-4163-b6ba-1df2e91bfcab.lab.blackmarble.co.uk:5985
  4. Make sure the server is set to use a shared UNC path for deployment.
  5. Remember you will login to this VM with the credentials for the test domain.


image

By this point you might be a bit confused as to what you have, well here is a diagram

image

Step 4  - Wiring the test into the pipeline

The final step is get the release pipeline to trigger the tests. This is done by calling the TCM.EXE command line to instruct the Test Controller trigger the tests. Now the copy of TCM does not have to be in Lab Management environment, but it does need to be on a VM known to RM vNext environment. This will usually mean a VM with Visual Studio Test Manager or Premium (or Enterprise for 2015) installed. In my case this was a dedicated test VM within the environment.

The key to the process is to run a script similar to the one used by the older RM agent based system to trigger the tests. You can extract this PowerShell script from an old release pipeline, but for ease I show my modified version here. The key changes are that I pass in the login credentials required for the call to the TFS server from TCM.EXE to be made from inside the network isolated environment and do a little extra checking of the test results so I can fail the build if the tests fail. These edits might not be required if you trigger TCM from a VM that is in the same domain as your TFS server, or have different success criteria.

param
(
    [string]$BuildDirectory = $null,
    [string]$BuildDefinition = $null,
    [string]$BuildNumber = $null,
    [string]$TestEnvironment = $null,
    [string]$LoginCreds = $null,
    [string]$Collection = $(throw "The collection URL must be provided."),
    [string]$TeamProject = $(throw "The team project must be provided."),
    [Int]$PlanId = $(throw "The test plan ID must be provided."),
    [Int]$SuiteId = $(throw "The test suite ID must be provided."),
    [Int]$ConfigId = $(throw "The test configuration ID must be provided."),
    [string]$Title = 'Automated UI Tests',
    [string]$SettingsName = $null,
    [Switch]$InconclusiveFailsTests = $false,
    [Switch]$RemoveIncludeParameter = $false,
    [Int]$TestRunWaitDelay = 10
)

 

##################################################################################
# Output the logo.
write-verbose "Based on the Microsoft Release Management TcmExec PowerShell Script v12.0"
write-verbose "Copyright (c) 2013 Microsoft. All rights reserved.`n"


 

##################################################################################
# Initialize the default script exit code.
$exitCode = 1

##################################################################################
# Output execution parameters.
write-verbose "Executing with the following parameters:"
write-verbose "  Build Directory: $BuildDirectory"
write-verbose "  Build Definition: $BuildDefinition"
write-verbose "  Build Number: $BuildNumber"
write-verbose "  Test Environment: $TestEnvironment"
write-verbose "  Collection: $Collection"
write-verbose "  Team project: $TeamProject"
write-verbose "  Plan ID: $PlanId"
write-verbose "  Suite ID: $SuiteId"
write-verbose "  Configuration ID: $ConfigId"
write-verbose "  Title: $Title"
write-verbose "  Settings Name: $SettingsName"
write-verbose "  Inconclusive result fails tests: $InconclusiveFailsTests"
write-verbose "  Remove /include parameter from /create command: $RemoveIncludeParameter"
write-verbose "  Test run wait delay: $TestRunWaitDelay"

##################################################################################
# Define globally used variables and constants.
# Visual Studio 2013
$vscommtools = [System.Environment]::GetEnvironmentVariable("VS120COMNTOOLS")
if ($vscommtools -eq $null)
{
    # Visual Studio 2012
    $vscommtools = [System.Environment]::GetEnvironmentVariable("VS110COMNTOOLS")
}
if ($vscommtools -eq $null)
{
    # Visual Studio 2010
    $vscommtools = [System.Environment]::GetEnvironmentVariable("VS100COMNTOOLS")
    if ($vscommtools -ne $null)
    {
        if ([string]::IsNullOrEmpty($BuildDirectory))
        {
            $(throw "The build directory must be provided.")
        }
        if (![string]::IsNullOrEmpty($BuildDefinition) -or ![string]::IsNullOrEmpty($BuildNumber))
        {
            $(throw "The build definition and build number parameters may be used only under Visual Studio 2012/2013.")
        }
    }
}
else
{
    if ([string]::IsNullOrEmpty($BuildDefinition) -and [string]::IsNullOrEmpty($BuildNumber) -and [string]::IsNullOrEmpty($BuildDirectory))
    {
        $(throw "You must specify the build directory or the build definition and build number.")
    }
}
$tcmExe = [System.IO.Path]::GetFullPath($vscommtools + "..\IDE\TCM.exe")

##################################################################################
# Ensure TCM.EXE is available in the assumed path.
if ([System.IO.File]::Exists($tcmExe))
{
    ##################################################################################
    # Prepare optional parameters.
    $testEnvironmentParameter = "/testenvironment:$TestEnvironment"
    if ([string]::IsNullOrEmpty($TestEnvironment))
    {
        $testEnvironmentParameter = [string]::Empty
    }
    if ([string]::IsNullOrEmpty($BuildDirectory))
    {
        $buildDirectoryParameter = [string]::Empty
    } else
    {
        # make sure we remove any trailing slashes as the cause permission issues
        $BuildDirectory = $BuildDirectory.Trim()
        while ($BuildDirectory.EndsWith("\"))
        {
            $BuildDirectory = $BuildDirectory.Substring(0,$BuildDirectory.Length-1)
        }
        $buildDirectoryParameter = "/builddir:""$BuildDirectory"""
   
    }
    $buildDefinitionParameter = "/builddefinition:""$BuildDefinition"""
    if ([string]::IsNullOrEmpty($BuildDefinition))
    {
        $buildDefinitionParameter = [string]::Empty
    }
    $buildNumberParameter = "/build:""$BuildNumber"""
    if ([string]::IsNullOrEmpty($BuildNumber))
    {
        $buildNumberParameter = [string]::Empty
    }
    $includeParameter = '/include'
    if ($RemoveIncludeParameter)
    {
        $includeParameter = [string]::Empty
    }
    $settingsNameParameter = "/settingsname:""$SettingsName"""
    if ([string]::IsNullOrEmpty($SettingsName))
    {
        $settingsNameParameter = [string]::Empty
    }

    ##################################################################################
    # Create the test run.
    write-verbose "`nCreating test run ..."
    $testRunId = & "$tcmExe" run /create /title:"$Title" /login:$LoginCreds /planid:$PlanId /suiteid:$SuiteId /configid:$ConfigId /collection:"$Collection" /teamproject:"$TeamProject" $testEnvironmentParameter $buildDirectoryParameter $buildDefinitionParameter $buildNumberParameter $settingsNameParameter $includeParameter
    if ($testRunId -match '.+\:\s(?<TestRunId>\d+)\.')
    {
        # The test run ID is identified as a property in the match collection
        # so we can access it directly by using the group name from the regular
        # expression (i.e. TestRunId).
        $testRunId = $matches.TestRunId

        write-verbose "Waiting for test run $testRunId to complete ..."
        $waitingForTestRunCompletion = $true
        while ($waitingForTestRunCompletion)
        {
            Start-Sleep -s $TestRunWaitDelay
            $testRunStatus = & "$tcmExe" run /list  /collection:"$collection" /login:$LoginCreds /teamproject:"$TeamProject" /querytext:"SELECT * FROM TestRun WHERE TestRunId=$testRunId"
            if ($testRunStatus.Count -lt 3 -or ($testRunStatus.Count -gt 2 -and $testRunStatus.GetValue(2) -match '.+(?<DateCompleted>\d+[/]\d+[/]\d+)'))
            {
                $waitingForTestRunCompletion = $false
            }
        }

        write-verbose "Evaluating test run $testRunId results..."
        # We do a small pause since the results might not be published yet.
        Start-Sleep -s $TestRunWaitDelay

        $testRunResultsTrxFileName = "TestRunResults$testRunId.trx"
        & "$tcmExe" run /export /id:$testRunId  /collection:"$collection" /login:$LoginCreds /teamproject:"$TeamProject" /resultsfile:"$testRunResultsTrxFileName" | Out-Null
        if (Test-path($testRunResultsTrxFileName))
        {
            # Load the XML document contents.
            [xml]$testResultsXml = Get-Content "$testRunResultsTrxFileName"
           
            # Extract the results of the test run.
            $total = $testResultsXml.TestRun.ResultSummary.Counters.total
            $passed = $testResultsXml.TestRun.ResultSummary.Counters.passed
            $failed = $testResultsXml.TestRun.ResultSummary.Counters.failed
            $inconclusive = $testResultsXml.TestRun.ResultSummary.Counters.inconclusive

            # Output the results of the test run.
            write-verbose "`n========== Test: $total tests ran, $passed succeeded, $failed failed, $inconclusive inconclusive =========="

            # Determine if there were any failed tests during the test run execution.
            if ($failed -eq 0 -and (-not $InconclusiveFailsTests -or $inconclusive -eq 0))
            {
                # Update this script's exit code.
                $exitCode = 0
            }

            # Remove the test run results file.
            remove-item($testRunResultsTrxFileName) | Out-Null
        }
        else
        {
            write-error "`nERROR: Unable to export test run results file for analysis."
        }
    }
}
else
{
    write-error "`nERROR: Unable to locate $tcmExe"
}

##################################################################################
# Indicate the resulting exit code to the calling process.
if ($exitCode -gt 0)
{
    write-error "`nERROR: Operation failed with error code $exitCode."
}
write-verbose "`nDone."
exit $exitCode

 

Once this script is placed into source control in such a way that it ends up in the drops location for the build you can call it as a standard script item in your pipeline, targeting the VM that has TCM installed. Remember, you get the test environment name and various IDs required from MTM. Check the TCM command line for more details.

image

 

However we hit a problem, RM sets PowerShell variable, not the parameters for script . So I find it easiest to use a wrapper script, also stored in source control, that converts the variable to the needed parameters. This also gives the opportunity to use RM set runtime variables and build more complex objects such as the credentials

 

# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose
$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition

write-verbose "Running $folder\TcmExecWithLogin.ps1"

& "$folder\TcmExecWithLogin.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -LoginCreds "$TestUserUid,$TestUserPwd" -SettingsName $SettingsName

 

Step 5 – Run it all

If you have everything in place you should now be able to trigger your deployment and have the tests run.

image

Finishing Up and One final gotcha

I had hoped that my integration test run would be associated with my build. Normally when triggering test via TCM you do this by adding the following parameters to the TCM command line

TCM [all the other params] -BuildNumber 'My.Build.CI_1.7.25.29773' -BuildDefinition 'My.Build.CI' 

However this will not work in the scenario above. This is because you can only use these flags to associate with successful builds, at the time TCM is run in the pipeline the build has not finished so it is not marked as successful. This does somewhat limit the end to end reporting. However, I think for now I can accept this limitation as the deployment completing is a suitable marker that the tests were passed.

The only workaround I can think is not to trigger the release directly from the build but to use the TFS events system to allow the build to finish first then trigger the release. You could use my TFS DSL Alert processor for that.

Stray white space in a ‘path to custom test adaptors’ will cause tests to fail on VSO vNext build

If you are providing a path to a custom test adaptor such as nUnit or Chutzpah for a TFS/VSO vNext build e.g. $(Build.SourcesDirectory)\packages, make sure you have no leading whitespace in the data entry form.

image

 

If you do have a space you will see an error log like this as the adaptor cannot be found as the command line generated is malformed

2015-07-13T16:11:32.8986514Z Executing the powershell script: C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\tasks\VSTest\1.0.16\VSTest.ps1
2015-07-13T16:11:33.0727047Z ##[debug]Calling Invoke-VSTest for all test assemblies
2015-07-13T16:11:33.0756512Z Working folder: C:\a\0549426d
2015-07-13T16:11:33.0777083Z Executing C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "C:\a\0549426d\UnitTestDemo\WebApp.Tests\Scripts\mycode.tests.js"  /TestAdapterPath: C:\a\0549426d\UnitTestDemo\Chutzpah /logger:trx
2015-07-13T16:11:34.3495987Z Microsoft (R) Test Execution Command Line Tool Version 12.0.30723.0
2015-07-13T16:11:34.3505995Z Copyright (c) Microsoft Corporation.  All rights reserved.
2015-07-13T16:11:34.3896000Z ##[error]Error: The /TestAdapterPath parameter requires a value, which is path of a location containing custom test adapters. Example:  /TestAdapterPath:c:\MyCustomAdapters
2015-07-13T16:11:36.5808275Z ##[error]Error: The test source file "C:\a\0549426d\UnitTestDemo\Chutzpah" provided was not found.
2015-07-13T16:11:37.0004574Z ##[error]VSTest Test Run failed with exit code: 1
2015-07-13T16:11:37.0094570Z ##[warning]No results found to publish.

    Cannot run Pester unit tests in Visual Studio but they work Ok from the command prompt

    I have been using Pester for some PowerShell tests. From the command prompt all is good, but I kept getting the error ‘module cannot be loaded because scripts is disabled on this system’ when I tried to run them via the Visual Studio Test Explorer

     

    image

    I found the solution on StackOverflow, I had forgotten that Visual Studio is 32bit, so you need to set the 32bit execution policy. Opening the default PowerShell command prompt and and setting the policy only effect the 64Bit instance.

    1. Open C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe
    2. Run the command Set-ExecutionPolicy RemoteSigned
    3. My tests passed (without restarting Visual Studio)

    image