Where did all my test results go?
Problem
I recently tripped myself up whist adding SonarQube analysis to a rather complex Azure DevOps build.
The build has two VsTest steps, both were using the same folder for their test result files. When the first VsTest task ran it created the expected .TRX and .COVERAGE files and then published its results to Azure DevOps, but when the second VsTest task ran it over wrote this folder, deleting the files already present, before it generated and published it results.
This meant that the build itself had all the test results published, but when SonarQube looked for the files for analysis only the second set of test were present, so its analysis was incorrect.
Solution
The solution was easy, use different folders for each set of test results.
This gave me a build, the key items are shown below, where one VsTest step does not overwrite the previous results before they can be processed by any 3rd party tasks such as SonarQube.
1steps:
2- task: SonarSource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@4
3 displayName: 'Prepare analysis on SonarQube'
4 inputs:
5 SonarQube: Sonarqube
6 projectKey: 'Services'
7 projectName: 'Services'
8 projectVersion: '$(major).$(minor)'
9 extraProperties: |
10 # Additional properties that will be passed to the scanner,
11 sonar.cs.vscoveragexml.reportsPaths=$(System.DefaultWorkingDirectory)/\*\*/\*.coveragexml
12 sonar.cs.vstest.reportsPaths=$(System.DefaultWorkingDirectory)/\*\*/\*.trx
13
14
15
16… other build steps
17
18
19
20\- task: VSTest@2
21 displayName: 'VsTest – Internal Services'
22 inputs:
23 testAssemblyVer2: |
24 \*\*\*.unittests.dll
25 !\*\*obj\*\*
26 searchFolder: '$(System.DefaultWorkingDirectory)/src/Services'
27 resultsFolder: '$(System.DefaultWorkingDirectory)TestResultsServices'
28 overrideTestrunParameters: '-DeploymentEnabled false'
29 codeCoverageEnabled: true
30 testRunTitle: 'Services Unit Tests'
31 diagnosticsEnabled: True
32 continueOnError: true
33
34\- task: VSTest@2
35 displayName: 'VsTest - External'
36 inputs:
37 testAssemblyVer2: |
38 \*\*\*.unittests.dll
39 !\*\*obj\*\*
40 searchFolder: '$(System.DefaultWorkingDirectory)/src/ExternalServices'
41 resultsFolder: '$(System.DefaultWorkingDirectory)TestResultsExternalServices'
42 vsTestVersion: 15.0
43 codeCoverageEnabled: true
44 testRunTitle: 'External Services Unit Tests'
45 diagnosticsEnabled: True
46 continueOnError: true
47
48\- task: BlackMarble.CodeCoverage-Format-Convertor-Private.CodeCoverageFormatConvertor.CodeCoverage-Format-Convertor@1
49 displayName: 'CodeCoverage Format Convertor'
50 inputs:
51 ProjectDirectory: '$(System.DefaultWorkingDirectory)'
52
53\- task: SonarSource.sonarqube.6D01813A-9589-4B15-8491-8164AEB38055.SonarQubeAnalyze@4
54 displayName: 'Run Code Analysis'
55
56\- task: SonarSource.sonarqube.291ed61f-1ee4-45d3-b1b0-bf822d9095ef.SonarQubePublish@4
57 displayName: 'Publish Quality Gate Result'