Running Microsoft Test Manager Test Suites as part of a vNext Release pipeline - Part 2

calendarAug 11, 2015 · 6 min read · Uncategorised  ·
Share on: linkedincopy

In my last post I discussed how you could wire TCM tests into a Release Management vNext pipeline. The problem with the script I provided, as I noted, was that the deployment was triggered synchronously by the build i.e. the build/release process was:

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Wait while the RM pipeline completed
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
  3. When RM completed the TFS build completes

This process raised a couple of problems

  • You cannot associate the integration tests with the build as TCM only allow association with completed successful builds. When TCM finishes in this model the build is still in progress.
  • You have to target only the first automated stage of the pipeline, else the build will be held as ‘in progress’ until all the release stages have complete, which may be days if there are manual approvals involved

The script InitiateReleaseFromBuild

These problems can all be fixed by altering the PowerShell that triggers the RM pipeline so that it does not wait for the deployment to complete, so the TFS build completes as soon as possible.

This is done by passing in an extra parameter which is set in TFS build

  1param(  
  2    \[string\]$rmserver = $Args\[0\],  
  3    \[string\]$port = $Args\[1\],    
  4    \[string\]$teamProject = $Args\[2\],     
  5    \[string\]$targetStageName = $Args\[3\],  
  6    \[string\]$waitForCompletion = $Args\[4\]  
  7) 
  8
  9cls  
 10$teamFoundationServerUrl = $env:TF\_BUILD\_COLLECTIONURI  
 11$buildDefinition = $env:TF\_BUILD\_BUILDDEFINITIONNAME  
 12$buildNumber = $env:TF\_BUILD\_BUILDNUMBER
 13
 14 
 15
 16  
 17"Executing with the following parameters:\`n"  
 18"  RMserver Name: $rmserver"  
 19"  Port number: $port"  
 20"  Team Foundation Server URL: $teamFoundationServerUrl"  
 21"  Team Project: $teamProject"  
 22"  Build Definition: $buildDefinition"  
 23"  Build Number: $buildNumber"  
 24"  Target Stage Name: $targetStageName\`n"  
 25"  Wait for RM completion: $waitForCompletion\`n"
 26
 27 
 28
 29$wait = \[System.Convert\]::ToBoolean($waitForCompletion)  
 30$exitCode = 0
 31
 32 
 33
 34trap  
 35{  
 36  $e = $error\[0\].Exception  
 37  $e.Message  
 38  $e.StackTrace  
 39  if ($exitCode -eq 0) { $exitCode = 1 }  
 40}
 41
 42 
 43
 44$scriptName = $MyInvocation.MyCommand.Name  
 45$scriptPath = Split-Path -Parent (Get-Variable MyInvocation -Scope Script).Value.MyCommand.Path
 46
 47 
 48
 49Push-Location $scriptPath    
 50
 51 
 52
 53$server = \[System.Uri\]::EscapeDataString($teamFoundationServerUrl)  
 54$project = \[System.Uri\]::EscapeDataString($teamProject)  
 55$definition = \[System.Uri\]::EscapeDataString($buildDefinition)  
 56$build = \[System.Uri\]::EscapeDataString($buildNumber)  
 57$targetStage = \[System.Uri\]::EscapeDataString($targetStageName)
 58
 59 
 60
 61$serverName = $rmserver + ":" + $port  
 62$orchestratorService = "[http://$serverName/account/releaseManagementService/\_apis/releaseManagement/OrchestratorService"](http://$serverName/account/releaseManagementService/_apis/releaseManagement/OrchestratorService")
 63
 64 
 65
 66$status = @{  
 67    "2" = "InProgress";  
 68    "3" = "Released";  
 69    "4" = "Stopped";  
 70    "5" = "Rejected";  
 71    "6" = "Abandoned";  
 72}
 73
 74 
 75
 76$uri = "$orchestratorService/InitiateReleaseFromBuild?teamFoundationServerUrl=$server&teamProject=$project&buildDefinition=$definition&buildNumber=$build&targetStageName=$targetStage"  
 77"Executing the following API call:\`n\`n$uri"
 78
 79 
 80
 81$wc = New-Object System.Net.WebClient  
 82$wc.UseDefaultCredentials = $true  
 83\# rmuser should be part rm users list and he should have permission to trigger the release.
 84
 85 
 86
 87#$wc.Credentials = new-object System.Net.NetworkCredential("rmuser", "rmuserpassword", "rmuserdomain")
 88
 89 
 90
 91try  
 92{  
 93    $releaseId = $wc.DownloadString($uri)
 94
 95 
 96
 97    $url = "$orchestratorService/ReleaseStatus?releaseId=$releaseId"
 98
 99 
100
101    $releaseStatus = $wc.DownloadString($url)
102
103 
104
105  
106    if ($wait -eq $true)  
107    {  
108        Write-Host -NoNewline "\`nReleasing ..."
109
110 
111
112        while($status\[$releaseStatus\] -eq "InProgress")  
113        {  
114            Start-Sleep -s 5  
115            $releaseStatus = $wc.DownloadString($url)  
116            Write-Host -NoNewline "."  
117        }
118
119 
120
121        " done.\`n\`nRelease completed with {0} status." -f $status\[$releaseStatus\]  
122    } else {
123
124 
125
126        Write-Host -NoNewline "\`nTriggering Release and exiting"  
127    }
128
129 
130
131}  
132catch \[System.Exception\]  
133{  
134    if ($exitCode -eq 0) { $exitCode = 1 }  
135    Write-Host "\`n$\_\`n" -ForegroundColor Red  
136}
137
138 
139
140if ($exitCode -eq 0)  
141{  
142    if ($wait -eq $true)  
143    {  
144        if ($releaseStatus -eq 3)  
145        {  
146          "\`nThe script completed successfully. Product deployed without error\`n"  
147        } else {  
148            Write-Host "\`nThe script completed successfully. Product failed to deploy\`n" -ForegroundColor Red  
149            $exitCode = -1 # reset the code to show the error  
150        }  
151    } else {  
152        "\`nThe script completed successfully. Product deploying\`n"  
153    }  
154}  
155else  
156{  
157  $err = "Exiting with error: " + $exitCode + "\`n"  
158  Write-Host $err -ForegroundColor Red  
159}
160
161 
162
163Pop-Location
164
165 
166
167exit $exitCode

The Script TcmExecWrapper

A change is also required in the wrapper script I use to trigger the TCM test run. We need to check the exit code from the inner TCM PowerShell script and update the TFS build quality appropriately.

To this I use the new REST API in TFS 2015 as this is far easier than using the older .NET client API. No DLLs to distribute.

It is worth noticing that

  • I pass the credentials into the script from RM that are used to talk to the TFS server. This is because I am running my tests in a network isolated TFS Lab Environment, this means I am in the wrong domain to see the TFS server without providing login details. If you are not working cross domain you could just use Default Credentials.
  • RM only passes the BuildNumber into the script e.g. MyBuild_1.2.3.4, but the REST API need the build id to set the quality. Hence the need for function Get-BuildDetailsByNumber to get the id from the name
 1\# Output execution parameters.  
 2$VerbosePreference ='Continue' # equiv to -verbose  
 3function Get-BuildDetailsByNumber  
 4{  
 5    param  
 6    (  
 7        $tfsUri ,  
 8        $buildNumber,  
 9        $username,   
10        $password  
11    )  
12    $uri = "$($tfsUri)/\_apis/build/builds?api-version=2.0&buildnumber=$buildNumber"  
13    $wc = New-Object System.Net.WebClient  
14    #$wc.UseDefaultCredentials = $true  
15    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)  
16      
17    write-verbose "Getting ID of $buildNumber from $tfsUri "  
18    $jsondata = $wc.DownloadString($uri) | ConvertFrom-Json   
19    $jsondata.value\[0\]  
20    
21}  
22function Set-BuildQuality  
23{  
24    param  
25    (  
26        $tfsUri ,  
27        $buildID,  
28        $quality,  
29        $username,   
30        $password  
31    )  
32    $uri = "$($tfsUri)/\_apis/build/builds/$($buildID)?api-version=1.0"  
33    $data = @{quality = $quality} | ConvertTo-Json  
34    $wc = New-Object System.Net.WebClient  
35    $wc.Headers\["Content-Type"\] = "application/json"  
36    #$wc.UseDefaultCredentials = $true  
37    $wc.Credentials = new-object System.Net.NetworkCredential($username, $password)  
38      
39    write-verbose "Setting BuildID $buildID to quality $quality via $tfsUri "  
40    $wc.UploadString($uri,"PATCH", $data)   
41      
42}  
43$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition  
44write-verbose "Running $folderTcmExecWithLogin.ps1"   
45& "$folderTcmExecWithLogin.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId  -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -LoginCreds "$TestUserUid,$TestUserPwd" -SettingsName $SettingsName -BuildNumber $BuildNumber -BuildDefinition $BuildDefinition  
46write-verbose "Got the exit code from the TCM run of $LASTEXITCODE"  
47$url = "$Collection/$Teamproject"  
48$jsondata = Get-BuildDetailsByNumber -tfsUri $url -buildNumber $BuildNumber -username $TestUserUid -password $TestUserPwd  
49$buildId = $jsondata.id  
50write-verbose "The build ID is $buildId"  
51$newquality = "Test Passed"  
52if ($LASTEXITCODE -gt 0 )  
53{  
54    $newquality = "Test Failed"  
55}  
56   
57write-verbose "The build quality is $newquality"  
58Set-BuildQuality -tfsUri $url  -buildID $buildId -quality $newquality -username $TestUserUid -password $TestUserPwd

Note: TcmExecWithLogin.ps1 is the same as in the In my last post

Summary

So with these changes the process is now

  1. TFS Build
    1. Gets the source
    2. Compiled the code
    3. Run the unit tests
    4. Trigger the RM pipeline
    5. Build ends
  2. RM then
    1. Deploys the code
    2. Runs the integration tests
    3. When the test complete we set the TFS build quality

This means we can associate both unit and integration tests with a build and target our release at any stage in the pipeline, it pausing at the points manual approval is required without blocking the initiating build.