Using Azure DevOps Migration Tools Again

I have recently been working with a client who needed to move Azure DevOps Work Items between Team Projects on different Azure DevOps instances. The only realistic choice was to use the free open source Azure DevOps Migration Tools.

I have used these tools before, but it was a while ago, and as it is under active development, I had to relearn a few things. It is fair to say that the learning curve for this tool is steep, but as the documentation does not hide this

WARNING: This tool is not designed for a novice. This tool was developed to support the scenarios below, and the edge cases that have been encountered by the 30+ contributors from around the Azure DevOps community. You should be comfortable with the TFS/Azure DevOps object model, as well as debugging code in Visual Studio

So where did I trip up this time?

But the documentation!

The documentation is OK, but can be confusing at the start as page links loop back on themselves. Adding to confusion is that there are details for older legacy V1 processors, current V1 processors and new V2 ones, but the reality is you are probably using only a small subset of the ones listed. Probably just current V1 ones for most use cases.

In my case I could ignore all bar the WorkItemMigrationConfigprocessor and the three 'beta' Test*MigrationConfig processors, and I suspect this is true for many people. This might not appear the case on a first read of the documentation.

Today, the WorkItemMigrationConfig processor is the one that does the bulk of the work, doing the work previously done by a set of legacy processors e.g creating Iteration and Area nodes.

Recreating Configuration Files

Always recreate your configuration file when you update the version of the migration tools you are using. The whole schema could have changed and some option settings certainly will have.

The documentation does tell you to do this, don't be tempted to just update the version number in the config file, it usually does not work as you miss a required new setting. It is invariably quicker to re-add in your source and target details, rather than work out what other flag has changed.

Watch our for case sensitivity

I found that the current version, V12, is much more case sensitive than previous versions. I initially had my target Team Project name in lower case, and the tools failed in the strangest way.

A connection was successfully made to the team projects, but when it tried to create the missing iteration or area paths it failed with message about being unable to create nodes i.e.

NewNode is not anchored in the target project, it cannot be created

A search of GitHub Issues, GitHub Discussions and Stack overflow thread for this error suggested problem with the configuration values for the NodeBasePaths settings. However, after a bit of interactive debugging, I found this was not the root cause. In fact, as I was doing a simple copy migration all tht was needed was an empty array for the NodeBasePaths, as I had first thought.

The debugging showed that the problem was the case of the target project name. The mismatch meant a regex expression was failing to match the built in root Area and Iteration nodes. Hence the error.

So, top tip, copy and paste in URLs and Team Project names to avoid stupid typos

How to increase the chances of a successful migration

I always start by migrating over a single work item of a given type into a test target project. When this works, I try a single work item of another type. I repeat this process until I have migrated all the work item types in use.

I will try to selectively pick sample work items so there are relationships between them, thus ensuring the linking is working, and ones with attachments, to ensure those are migrated correctly too.

Once I know all the work item types can be migrated, then and only then do i try to run a migration for all the work items in the source project.

It is far better to spend a few minutes making sure all the work item type field and states are mapped correctly, rather than finding a few hours into a long migration run.

The way I do this filtering is to edit the WIQLQueryBit

1"WIQLQueryBit": "AND  [Microsoft.VSTS.Common.ClosedDate] = '' AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request') AND [System.Id] = '1234'",

Note: The configuration file does contain a "WorkItemIDs":[] setting that the documentation says is "A list of work items to import". I had assumed this would filter the work items returned by the WIQL query, it does not. After looking at the code it does not seem to be used at present in the WorkItemMigrationConfig processor.

A key advantage of the WIQL approach is by filtering the number of work items returned to a single one, the migration tools will start much more quickly. There is a 'tax' on every run of the migration tools where it has to do analysis of the source and target projects to work out what needs to be migrated. If it has to check hundreds or thousands of work items this will take many minutes. However, if it only has to check one, it will be done in, from my experience, around 10 seconds

Summary

Using this tool is always an adventure, but it does what it says it will when you have it configured correctly.

And if you do have problems, the debugging logs are detailed, and there is active support on GitHub Issues and GitHub Discussions.

If you do have problems, don't be afraid of running the tool using Visual Studio. This is a great way to debug the flow and see what is going on. If you do make changes remember this is an open source project and your bug fix and enhancement PRs will always be considered.

The bottom line is that you just have to accept that this form of migration will be a slow process, but once you have the basics of the configuration file correct the tool is reliable.