Updated Reprint - Migrating a TFS TFVC team project to a Git team project
This is a copy of the guest post done on the Microsoft UK web site published on the 7th June 2016
This is a revised version of a post originally published in August 2014. In this revision I have updated version numbers and links for tools used and added a discussion of adapting the process to support VSTS.
The code for this post can be found in my GitHub Repo
In the past I've written on the theory behind migrating TFVC to Git with history. I've since used this process for real, as opposed to as a proof of concept, and this post documents my experiences. The requirement was to move an on-premises TFS 2013.2 Scrum Team Project using TFVC to another on premises TFS 2013.2 Scrum Team Project, but this time using Git.
This process is equally applicable to any version of TFS that supports Git, and to VSTS.
Create new team project
On the target server create a new team project using the same (or as close as possible) process template as was used on the source TFS server. As we were using the same non-customised process template for both the source and the target we did not have to worry over any work item customisation. However, if you were changing the process template, this is where you would do any customisation required.
Remember that if you are targeting VSTS your customisation options are limited. You canadd custom fields to VSTS as of the time of writing (May 2016), but that is all.
Adding a field to all Work Item Types
We need to be able to associate the old work item ID with the new migrated one. For on-premises TFS servers, the TFS Integration Platform has a feature to do this automatically, but it suffers a bug. It is meant to automatically add a field for this purpose, but it actually needs it to be manually added prior to the migration.
To do this edit we need to either:
- Edit the process templates in place using the Process Template Editor Power Tool
- Export the WIT with WITADMIN.exe and edit them in Notepad and re-import them
In either case the field to add to ALL WORK ITEM TYPES is as follows:
1<FIELD refname="TfsMigrationTool.ReflectedWorkItemId" name="ReflectedWorkItemId" type="String">
Once the edit is made the revised work item types need to be re-imported back into the new Team project.
If you are using VSTS this way of adding the field is not an option, but we can add custom fields to a work item type to VSTS. If we do this you will need to use the TFS Integration Mapper tool (mentioned below) to make sure the required old work item ID ends up in your custom location. TFS Integration Platform will not do this by default, butI have documented this process in an associated post.
The Work Item Migration
The actual work item migration is done using the TFS Integration Platform. This tool says it only supports TFS 2012, but it will function with newer versions of TFS as well as VSTS. This will move over all work item types from the source team project to the target team project. The process is as follows:
- Install TFS Integration Platform.
- Load TFS Integration Platform, as it seems it must be loaded after the team project is created, else it gets confused!
- Select 'Create New'.
- Pick the 'Team Foundation ServerWorkItemTracking' template. As we are migrating with the same process template this is OK. If you need to change field mappings use the template for field matching and look at the TFS Integration Mapper tool.
- Provide a sensible name for the migration. Not really needed for a one-off migration, but if testing, it's easy to end up with many test runs all of the same name, which is confusing in the logs.
- Pick the source server and team project as the left server.
- Pick the target server and team project as the right server.
- Accept the defaults and save to database.
- On the left menu select Start. The UI on this tool is not great. Avoid looking on the output tab as this seems to slow the process. Also, altering the refresh time on the options for once a minute seems to help process performance. All details of actions are placed in log files so nothing is lost by these changes.
- The migration should complete without any issues, assuming there are no outstanding template issues that need to be resolved.
Add the New ID to the Changesets on the source server
The key to this migration process to retain the links between the work items and source code checkins. This is done using the technique I outlined in the previous post i.e. editing the comments field of the changeset on the source team project prior to migration the source, adding #123 style references to point to the new work items on the target server.
To do this I used some PowerShell. This PowerShell was written before the new TFS REST API was available, hence uses the older C# API. If I was writing it now I would have used the REST API.
1
2
3function Update-TfsCommentWithMigratedId
4{
5
6<#
7.SYNOPSIS
8This function is used as part of the migration for TFVC to Git to help retain checkin associations to work items
9
10.DESCRIPTION
11This function takes two team project references and looks up changset association in the source team project, it then looks for
12the revised work itme IT in the new team project and updates the source changeset
13
14.PARAMETER SourceCollectionUri
15Source TFS Collection URI
16
17.PARAMETER TargetCollectionUri
18Target TFS Collection URI
19
20.PARAMETER SourceTeamProject
21Source Team Project Name
22
23.EXAMPLE
24
25Update-TfsCommentWithMigratedId -SourceCollectionUri "[http://server1:8080/tfs/defaultcollection"](http://server1:8080/tfs/defaultcollection") -TargetCollectionUri "[http://server2:8080/tfs/defaultcollection"](http://server2:8080/tfs/defaultcollection") -SourceTeamProject "Scrumproject"
26
27#>
28
29 Param
30 (
31 \[Parameter(Mandatory=$true)\]
32 \[uri\] $SourceCollectionUri,
33
34 \[Parameter(Mandatory=$true)\]
35 \[uri\] $TargetCollectionUri,
36
37 \[Parameter(Mandatory=$true)\]
38 \[string\] $SourceTeamProject
39
40 )
41
42 # get the source TPC
43 $sourceTeamProjectCollection = New-Object Microsoft.TeamFoundation.Client.TfsTeamProjectCollection($sourceCollectionUri)
44 # get the TFVC repository
45 $vcService = $sourceTeamProjectCollection.GetService(\[Microsoft.TeamFoundation.VersionControl.Client.VersionControlServer\])
46 # get the target TPC
47 $targetTeamProjectCollection = New-Object Microsoft.TeamFoundation.Client.TfsTeamProjectCollection($targetCollectionUri)
48 #Get the work item store
49 $wiService = $targetTeamProjectCollection.GetService(\[Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore\])
50
51 # Find all the changesets for the selected team project on the source server
52 foreach ($cs in $vcService.QueryHistory(”$/$SourceTeamProject”, \[Microsoft.TeamFoundation.VersionControl.Client.RecursionType\]::Full, \[Int32\]::MaxValue))
53 {
54 if ($cs.WorkItems.Count -gt 0)
55 {
56 foreach ($wi in $cs.WorkItems)
57 {
58 "Changeset {0} linked to workitem {1}" -f $cs.ChangesetId, $wi.Id
59 # find new id for each changeset on the target server
60 foreach ($newwi in $wiService.Query("select id FROM WorkItems WHERE \[TfsMigrationTool.ReflectedWorkItemId\] = '" + $wi.id + "'"))
61 {
62 # if ID found update the source server if the tag has not already been added
63 # we have to esc the \[ as gets treated as a regular expression
64 # we need the white space around between the \[\] else the TFS agent does not find the tags
65 if ($cs.Comment -match "\[ Migrated ID #{0} \]" -f $newwi.Id)
66 {
67 Write-Output ("New Id {0} already associated with changeset {1}" -f $newwi.Id , $cs.ChangesetId)
68 } else {
69 Write-Output ("New Id {0} being associated with changeset {1}" -f $newwi.Id, $cs.ChangesetId )
70 $cs.Comment += "\[ Migrated ID #{0} \]" -f $newwi.Id
71 }
72 }
73 }
74 $cs.Update()
75 }
76 }
77}
78
With the usage:
1Update-TfsCommentWithMigratedId -SourceCollectionUri "http://localhost:8080/tfs/defaultcollection" -TargetCollectionUri "http://localhost:8080/tfs/defaultcollection" -SourceTeamProject "Old team project"
NOTE: This script is written so that it can be run multiple times, but only adds the migration entries once for any given changeset. This means both it and TFS Integration Platform can be run repeatedly on the same migration to do a staged migration e.g. get the bulk of the content over first whilst the team is using the old team project, then do a smaller migration of the later changes when the actual swap over happens.
When this script is run expect to see output similar to:
You can see the impact of the script in Visual Studio Team Explorer or the TFS web client when looking at changesets in the old team project. Expect to see a changeset comment in the form shown below with new [ Migrated ID #123 ] blocks in the comment field, with 123 being the work item ID on the new team project. Also note the changeset is still associated with the old work item ID on the source server.
NOTE: The space after the #123 is vital. If it is not there, then the TFS job agent cannot find the tag to associate the commit to a work item after the migration.
Source code migration
The source code can now be migrated. This is done by cloning the TFVC code to a local Git repo and then pushing it up to the new TFS Git repo using Git TF. We clone the source to a local repo in the folder localrepo with the -deep option used to retain history.
1git tf clone http://typhoontfs:8080/tfs/defaultcollection '$/Scrum TFVC Source/Main' localrepo --deep
NOTE: I have seen problems with this command. On larger code bases we saw the error 'TF 400732 server cancelled error' as files were said to be missing or we had no permission - neither of which was true. This problem was repeated on a number of machines, including one that had in the past managed to do the clone. It was thought the issue was on the server connectivity, but no errors were logged.
As a work around the Git-TFS tool was used. This community tool uses the .NET TFS API, unlike the Microsoft one which uses the Java TFS API. Unfortunately, it also gave TF400732 errors, but did provide a suggested command line to retry continue, which continued from where it errored.
The command to do the clone was:
1Git tfs clone http://typhoontfs:8080/tfs/defaultcollection '$/Scrum TFVC Source/Main' localrepo
The command to continue after an error was (from within the repo folder):
1Git tfs fetch
It should be noted that Git-TFS seems a good deal faster than Git TF, presumably due to being a native .NET client as opposed to using the Java VM. Also, Git-TFS has support for converting TFVC branches to Git branches, something Git TF is not able to do. So for some people, Git-TFS will be a better tool to use.
Once the clone is complete, we need to add the TFS Git repo as a remote target and then push the changes up to the new team project. The exact commands for this stage are shown on the target TFS server. Load the web client, go to the code section and you should see the commands needed:
1git remote add origin http://typhoontfs:8080/tfs/DefaultCollection/\_git/newproject git push -u origin --all
Once this stage is complete the new TFS Git repo can be used. The Git commits should have the correct historic date and work item associations as shown below. Note now that the migration ID comments match the work item associations.
NOTE: There may be a lack in the associations being shown immediately after the git push. This is because the associations are done by a background TFS job process which may take a while to catch up when there are a lot of commits. On one system I worked on this took days, not hours! Be patient.
Shared Test Steps
At this point all work items have been moved over and their various associations with source commits are retained e.g. PBIs link to test cases and tasks. However, there is a problem that any test cases that have shared steps will be pointing to the old shared step work items. As there is already an open source tool to do this update, there was no immediate need to rewrite it as a PowerShell tool. So to use the open source tool use the command line:
1UpdateSharedStep.exe http://localhost:8080/tfs/defaultcollection myproject
Test Plans and Suites
Historically in TFS, test plans and suites were not work items, they became work items in TFS 2013.3. This means if you need these moved over too, then you had to use the TFS API.
Though these scripts were written for TFS 2013.2, there is no reason for these same API calls not to work with newer versions of TFS or VSTS. Just remember to make sure you exclude the Test Plans and Suites work items from the migration performed TFS Integration Platform so you don't move them twice.
This script moves the three test suite types as follows:
- Static - Creates a new suite, finds the migrated IDs of the test cases on the source suite and adds them to the new suite.
- Dynamic - Creates a new suite using the existing work item query. IMPORTANT - The query is NOT edited, so it may or may not work depending on what it actually contained. These suites will need to be checked by a tester manually in all cases and their queries 'tweaked'.
- Requirements - Create a new suite based on the migrated IDs of the requirement work items. This is the only test suite type where we edit the name to make it consistent with the new requirement ID not the old.
The script is as follows:
1function Update-TestPlanAfterMigration
2{
3<#
4.SYNOPSIS
5This function migrates a test plan and all its child test suites to a different team project
6
7.DESCRIPTION
8This function migrates a test plan and all its child test suites to a different team project, reassign work item IDs as required
9
10.PARAMETER SourceCollectionUri
11Source TFS Collection URI
12
13.PARAMETER SourceTeamProject
14Source Team Project Name
15
16.PARAMETER SourceCollectionUri
17Target TFS Collection URI
18
19.PARAMETER SourceTeamProject
20Targe Team Project Name
21
22
23.EXAMPLE
24
25Update-TestPlanAfterMigration -SourceCollectionUri "[http://server1:8080/tfs/defaultcollection"](http://server1:8080/tfs/defaultcollection") -TargetCollectionUri "[http://serrver2:8080/tfs/defaultcollection"](http://serrver2:8080/tfs/defaultcollection") -SourceTeamProjectName "Old project" -TargetTeamProjectName "New project"
26
27#>
28 param(
29 \[Parameter(Mandatory=$true)\]
30 \[uri\] $SourceCollectionUri,
31
32 \[Parameter(Mandatory=$true)\]
33 \[string\] $SourceTeamProjectName,
34
35 \[Parameter(Mandatory=$true)\]
36 \[uri\] $TargetCollectionUri,
37
38 \[Parameter(Mandatory=$true)\]
39 \[string\] $TargetTeamProjectName
40
41 )
42
43 # Get TFS connections
44 $sourcetfs = \[Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory\]::GetTeamProjectCollection($SourceCollectionUri)
45 try
46 {
47 $Sourcetfs.EnsureAuthenticated()
48 }
49 catch
50 {
51 Write-Error "Error occurred trying to connect to project collection: $\_ "
52 exit 1
53 }
54 $targettfs = \[Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory\]::GetTeamProjectCollection($TargetCollectionUri)
55 try
56 {
57 $Targettfs.EnsureAuthenticated()
58 }
59 catch
60 {
61 Write-Error "Error occurred trying to connect to project collection: $\_ "
62 exit 1
63 }
64
65 # get the actual services
66 $sourcetestService = $sourcetfs.GetService("Microsoft.TeamFoundation.TestManagement.Client.ITestManagementService")
67 $targettestService = $targettfs.GetService("Microsoft.TeamFoundation.TestManagement.Client.ITestManagementService")
68 $sourceteamproject = $sourcetestService.GetTeamProject($sourceteamprojectname)
69 $targetteamproject = $targettestService.GetTeamProject($targetteamprojectname)
70 # Get the work item store
71 $wiService = $targettfs.GetService(\[Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore\])
72
73
74 # find all the plans in the source
75 foreach ($plan in $sourceteamproject.TestPlans.Query("Select \* From TestPlan"))
76 {
77 if ($plan.RootSuite -ne $null -and $plan.RootSuite.Entries.Count -gt 0)
78 {
79 # copy the plan to the new tp
80 Write-Host("Migrating Test Plan - {0}" -f $plan.Name)
81 $newplan = $targetteamproject.TestPlans.Create();
82 $newplan.Name = $plan.Name
83 $newplan.AreaPath = $plan.AreaPath
84 $newplan.Description = $plan.Description
85 $newplan.EndDate = $plan.EndDate
86 $newplan.StartDate = $plan.StartDate
87 $newplan.State = $plan.State
88 $newplan.Save();
89 # we use a function as it can be recursive
90 MoveTestSuite -sourceSuite $plan.RootSuite -targetSuite $newplan.RootSuite -targetProject $targetteamproject -targetPlan $newplan -wiService $wiService
91 # and have to save the test plan again to persit the suites
92 $newplan.Save();
93
94 }
95 }
96
97
98
99}
100
101\# - is missing in name so this method is not exposed when module loaded
102function MoveTestSuite
103{
104<#
105.SYNOPSIS
106This function migrates a test suite and all its child test suites to a different team project
107
108.DESCRIPTION
109This function migrates a test suite and all its child test suites to a different team project, it is a helper function Move-TestPlan and will probably not be called directly from the command line
110
111.PARAMETER SourceSuite
112Source TFS test suite
113
114.PARAMETER TargetSuite
115Target TFS test suite
116
117.PARAMETER TargetPlan
118The new test plan the tests suite are being created in
119
120.PARAMETER targetProject
121The new team project test suite are being created in
122
123.PARAMETER WiService
124Work item service instance used for lookup
125
126
127.EXAMPLE
128
129Move-TestSuite -sourceSuite $plan.RootSuite -targetSuite $newplan.RootSuite -targetProject $targetteamproject -targetPlan $newplan -wiService $wiService
130
131#>
132 param
133 (
134 \[Parameter(Mandatory=$true)\]
135 $sourceSuite,
136
137 \[Parameter(Mandatory=$true)\]
138 $targetSuite,
139
140 \[Parameter(Mandatory=$true)\]
141 $targetProject,
142
143 \[Parameter(Mandatory=$true)\]
144 $targetplan,
145
146 \[Parameter(Mandatory=$true)\]
147 $wiService
148 )
149
150 foreach ($suite\_entry in $sourceSuite.Entries)
151 {
152 # get the suite to a local variable to make it easier to pass around
153 $suite = $suite\_entry.TestSuite
154 if ($suite -ne $null)
155 {
156 # we have to build a suite of the correct type
157 if ($suite.IsStaticTestSuite -eq $true)
158 {
159 Write-Host(" Migrating static test suite - {0}" -f $suite.Title)
160 $newsuite = $targetProject.TestSuites.CreateStatic()
161 $newsuite.Title = $suite.Title
162 $newsuite.Description = $suite.Description
163 $newsuite.State = $suite.State
164 # need to add the suite to the plan else you cannot add test cases
165 $targetSuite.Entries.Add($newSuite) >$nul # sent to null as we get output
166 foreach ($test in $suite.TestCases)
167 {
168 $migratedTestCaseIds = $targetProject.TestCases.Query("Select \* from \[WorkItems\] where \[TfsMigrationTool.ReflectedWorkItemId\] = '{0}'" -f $Test.Id)
169 # we assume we only get one match
170 if ($migratedTestCaseIds\[0\] -ne $null)
171 {
172 Write-Host (" Test {0} has been migrated to {1} and added to suite {2}" -f $Test.Id , $migratedTestCaseIds\[0\].Id, $newsuite.Title)
173 $newsuite.Entries.Add($targetProject.TestCases.Find($migratedTestCaseIds\[0\].Id)) >$nul # sent to null as we get output
174 }
175 }
176 }
177
178
179 if ($suite.IsDynamicTestSuite -eq $true)
180 {
181 Write-Host(" Migrating query based test suite - {0} (Note - query may need editing)" -f $suite.Title)
182 $newsuite = $targetProject.TestSuites.CreateDynamic()
183 $newsuite.Title = $suite.Title
184 $newsuite.Description = $suite.Description
185 $newsuite.State = $suite.State
186 $newsuite.Query = $suite.Query
187
188 $targetSuite.Entries.Add($newSuite) >$nul # sent to null as we get output
189 # we don't need to add tests as this is done dynamically
190
191 }
192
193 if ($suite.IsRequirementTestSuite -eq $true)
194 {
195 $newwis = $wiService.Query("select \* FROM WorkItems WHERE \[TfsMigrationTool.ReflectedWorkItemId\] = '{0}'" -f $suite.RequirementId)
196 if ($newwis\[0\] -ne $null)
197 {
198 Write-Host(" Migrating requirement based test suite - {0} to new requirement ID {1}" -f $suite.Title, $newwis\[0\].Id )
199
200 $newsuite = $targetProject.TestSuites.CreateRequirement($newwis\[0\])
201 $newsuite.Title = $suite.Title -replace $suite.RequirementId, $newwis\[0\].Id
202 $newsuite.Description = $suite.Description
203 $newsuite.State = $suite.State
204 $targetSuite.Entries.Add($newSuite) >$nul # sent to null as we get output
205 # we don't need to add tests as this is done dynamically
206 }
207 }
208
209 # look for child test cases
210 if ($suite.Entries.Count -gt 0)
211 {
212 MoveTestSuite -sourceSuite $suite -targetSuite $newsuite -targetProject $targetteamproject -targetPlan $newplan -wiService $wiService
213 }
214 }
215 }
216 }
217
NOTE: This script needs PowerShell 3.0 installed. This appears to be because some the TFS assemblies are .NET 4.5 which is not supported by previous PowerShell versions. If the version is wrong the test suite migration will fail as the TestPlan (ITestPlanHelper) object will be null.
The command to run the migration of test plans is:
1Update-TestPlanAfterMigration -SourceCollectionUri "http://typhoontfs:8080/tfs/defaultcollection" -TargetCollectionUri "http://typhoontfs:8080/tfs/defaultcollection" -SourceTeamProjectName "Scrum TFVC Source" -TargetTeamProjectName "NewProject"
This will create the new set of test plans and suites in addition to any already in place on the target server. It should give an output similar to:
Summary
Once all this is done you should have migrated a TFVC team project to a new team project based on Git on either on-premises TFS or VSTS, retaining as much history as is possible. I hope you find this of use!
This article was first published on the Microsoft’s UK Developers site Migrating a TFS TFVC based team project to a Git team project - a practical example originally published August the 15th 2014 updated 7 June 2016