Strange build issue in Azure DevOps Pipelines when MSBuild is running pre-build scripts

The Problem

I recently came across a problem with a complex BitTalk solution being built in an Azure DevOps Pipeline. It had been working fine, but it started to fail with error related to failing to copy built DLLs from the obj folder to the bin folder because it said the files were in use by MSBuild.

The Analysis

I tried a few things, all to little effect:

  • Restarted the self-hosted Azure DevOps Pipeline agent, in case they were holding a lock on a file
  • Delete the builds _work\<number> folder, in case it had become corrupted
  • Re-ran the build against the last Git commit, this worked so at least proving the issue was in a code change and not something to do with the build agent VM.
  • Built the solution locally on a development PC via Visual Studio, it built OK
  • Built the solution using the same, or at least similar (removing the Azure DevOps specific flags) MSBuild command locally on a development PC, and this time I got the same error.

So the issue was a recent code changes, but only showed up when using MSBuild directly, but not when letting Visual Studio orchestrate the build process.

The Solution

The recent change was that pre-build PowerShell has been added to a number of project files to register 3rd party assemblies in the Windows GAC. This change was proved to be the issues as if I commented out this pre-build step the local MSBuild test build completed without an error.

It turns out the problem was the pre-build step had been added to many projects in the solution, thus causing race conditions. The fix was to make sure we only registered the assemblies in the GAC once in the whole solution, in the first project to be built.

This all makes sense, but I did think it strange it showed up initially as what appeared to be an unrelated file copy/lock error.

For the original version of this post see Richard Fennell's personal blog at Strange build issue in Azure DevOps Pipelines when MSBuild is running pre-build scripts