Running WebTests as part of a VSTS VNext Release pipeline

author Richard Fennell calendar May 30, 2016 · 4 min read · Uncategorised  ·
Share on: linkedin copy

Background

Most projects will have a range of tests

  • Unit tests (maybe using a mocking framework) running inside the build process
  • Integration/UX and load tests run as part of a release pipeline
  • and finally manual tests

In a recent project we were using WebTests to provide some integration tests (in addition to integration tests written using unit testing frameworks) as a means to test a REST/ODATA API, injecting data via the API, pausing while a backend Azure WebJob processed the injected data, then checking a second API to make sure the processed data was correctly presented. Basically mimicking user operations.

In past iterations we ran these tests via TFS Lab Management’s tooling, using the Test Agent that is deploys when an environment is created.

The problem was we are migrating to VSTS/TFS 2015.2 Release Management. This uses the new Functional Testing Task, which uses the newer Test Agent that is deployed on demand as part of the release pipeline (not pre-installed) and this agent does not support running WebTests at present.

This means my only option was to use MsTest if I wanted to continue using this form of webtest. However, there is no out the box MsTest task for VSTS, so I needed to write a script to do the job that I could deploy as part of my build artifacts.

Now I could write a build/release task to make this nice and easy to use, but that is more work and I suspect that I am not going to need this script too often in the future (I might be wrong here only time will tell). Also I hope that Microsoft will at some point provide an out the box task to do the job either by providing an MStest task or adding webtest support to the functional test task.

This actually reflects my usual work practice for build tasks, get the script working first locally, use it as PowerShell script in the build, and if I see enough refuse make it a task/extension.

So what did I actually need to do?

Preparation

  1. Install Visual Studio on the VM where the tests will be run from. I need to do this because though MSTest was already present  it fails to run .Webtest tests unless a suitable SKU of Visual Studio is installed

  2. Set the solution configuration so that the projects containing the webtests is not built, we only need the .webtest files copied to the drops location. If you build the project the files get duplicated into the bin folder, which we don’t need as we then need to work out which copy to use.

  3. Make sure the solution contains a .TestSettings file that switches on ‘Think Times’, and this file is copied as a build artifact. This stalled me for ages, could not work out why tests worked in Visual Studio and failed from the command line. Without this file there is no think time at all so my background process never had time to run.

    image

  4. Write a script that finds all my .Webtest files and place it in source control such that it is copied to the builds drop location.

 1param 
 2
 3(
 4
 5    $tool = "C:Program Files (x86)Microsoft Visual Studio 14.0Common7IDEMSTest.exe",  
 6    $path ,  
 7    $include = "\*.webtest",  
 8    $results ,  
 9    $testsettings
10
11)
12
13 
14
15$web\_tests = get-ChildItem -Path $paths -Recurse -Include $include
16
17foreach ($item in $web\_tests) {  
18    $args += "/TestContainer:$item "
19
20}
21
22  
23& $tool $args /resultsfile:$Results /testsettings:$testsettings

Build

Once the script and other settings are made I altered the build so that the .webtests (including their associated JSON test data sub folders), the script and the .testsettings files are all copied to the drops location

image

Release

In the release pipeline I need to call my script with suitable parameters so it find the tests, uses the .testsettings and creates a .TRX results file. I then need to use the ‘Publish Test Results’ task to uploaded these MSTest format results

image

So for the PowerShell MSTest task I set the following

  • Script name is $(System.DefaultWorkingDirectory)/MyBuilddropScriptsRunMSTest.ps1 
  • The argument is -path $(System.DefaultWorkingDirectory)MyBuilddropSrcWebtestsProject -results $(System.DefaultWorkingDirectory)webtests.trx -testsettings $(System.DefaultWorkingDirectory)MyBuilddropsrcwebtest.testsettings

And for the publish test results task.

  • Format – VSTest
  • Arguments - $(System.DefaultWorkingDirectory)webtests.trx
  • I also set this task to always run to make sure I got test results even if some test failed

Once all this was done and the build/release run I got my test results I needed

image

I can drill into my detailed test reports as needed

image

So I have a functioning release pipeline that can run all the various types of automated tests within my solution.