Whilst running tests as part of a Release Management deployment I started seeing the error ‘The build directory of the test run either does not exist or access permission is required’, and hence all my tests failed. It seems that there are issues that can cause this problem, as mentioned in the comments in Martin Hinshelwood’s post on running tests in deployment, specially spaces in the build name can cause this problem, but this was not the case for me.
Strangest point was it used to work, what had I changed?
To debug the problem I logged into the test VM as the account the deployment service was running as (a shadow account as the environment was network isolated). I got the command line that the component was trying to run by looking at the messages in the deployment log
I then went to the deployment folder on the test VM
%appdata%\local\temp\releasemanagement\[the release management component name]\[release number]
and ran the same command line. Strange thing was this worked! All the tests ran and passed OK, TFS was updated, everything was good.
It seemed I only had an issue when triggering the tests via a Release Management deployment, very strange!
A side note here, when I say the script ran OK it did report an error and did not export and unpack the test results from the TRX file to pass back to the console/release management log. Turns out this is because the MTMExec.ps1 script uses the command [System.IO.File]::Exist(..) to check if the .TRX file has been produced. This fails when the script is run manually. This is because it relies on [Environment]::CurrentDirectory, which is not set the same way when run manually as when a script is called by the deployment service. When run manually it seems to default to c:\windows\system32 not the current folder.
If you are editing this script, and want it to work in both scenarios, then probably best to use the PowerShell Test-Path(..) cmdlet as opposed to [System.IO.File]::Exist(..)
So where to look for this problem, the error says something can’t access the drops location, but what?
A bit of thought as to who is doing what can help here
When the deployment calls for a test to be run
- The Release Management deployment agent pulls the component down to the test VM from the Release Management Server
- It then runs the Powershell Script
- The PowerShell script runs TCM.exe to trigger the test run, passing in the credentials to access the TFS server and Test Controller
- The Test Controller triggers the tests to be run on the Test Agent, providing it with the required DLLs from the TFS drops location – THIS IS THE STEP WITH THE PROBLEM IS SEEN
- The Test Agent runs the tests and passes the results back to TFS via the Test Controller
- After the PowerShell script triggers the test run it loops until the test run is complete.
- It then uses TCM again to extract the test results, which it parses and passes back to the Release Management server
So a good few places to check the logs.
Turns out the error was being reported on the Test Controller.
(QTController.exe, PID 1208, Thread 14) Could not use lab service account to access the build directory. Failure: Network path does not exist or is not accessible using following user: \\store\drops\Sabs.Main.CI\Sabs.Main.CI_220.127.116.1138\ using blackmarble\tfslab. Error Code: 53
The error told me the folder and who couldn’t access it, the domain service account ‘tfslab’ the Test Agents use to talk back to the Test Controller.
I checked the drops location share and this user has adequate access rights. I even logged on to the Test Controller as this user and confirmed I could open the share.
I then had a thought, this was the account the Test Agents were using to communicate with the Test Controller, but was it the account the controller was running as? A check showed it was not, the controller was running as the default ‘Local System’. As soon as I swapped to using the lab service account (or I think any domain account with suitable rights) it all started to work.
So why did this problem occur?
All I can think of was that (to address another issue with Windows 8.1 Coded-UI testing) the Test Controller was upgraded to 2013.2RC, but the Test Agent in this lab environment was still at 2013RTM. Maybe the mismatch is the issue?
I may revisit and retest with the ‘Local System’ account when 2013.2 RTM’s and I upgrade all the controllers and agents, but I doubt it. I have no issue running the test controller as a domain account.