Fixing a ‘git-lfs filter-process: gif-lfs: command not found’ error in Visual Studio 2017

I am currently looking at the best way to migrate a large legacy codebase from TFVC to Git. There are a number of ways I could do this, as I have posted about before. Obviously, I have ruled out anything that tries to migrate history as ‘that way hell lies’; if people need to see history they will be able to look at the archived TFVC instance. TFVC and Git are just too different in the way they work to make history migrations worth the effort in my opinion.

So as part of this migration and re-structuring I am looking at using Git Submodules and Git Large File System (LFS) to help divide the monolithic code base into front-end, back-end and shared service modules; using LFS to manage large media files used in integration test cases.

From the PowerShell command prompt, using Git 2.16.2, all my trials were successful, I could achieve what I wanted. However when I tried accessing my trial repos using Visual Studio 2017 I saw issues

Submodules

Firstly there are known limitations with Git submodules in Visual Studio Team Explorer. At this time you can clone a repo that has submodules, but you cannot manage the relationships between repos or commit to a submodule from inside Visual Studio.

This is unlike the Git command line, which allows actions to span a parent and child repo with a single command, Git just works it out if you pass the right parameters

There is a request on UserVoice to add these functions to Visual Studio, vote for it if you think it is important, I have.

Large File System

The big problem I had was with LFS, which is meant to work in Visual Studio since 2015.2.

Again from the command line operations were seamless, I just installed Git 2.16.2 via Chocolaty and got LFS support without installing anything else. So I was able to enable LFS support on a repo

git lfs install
git lfs track '*.bin'
git add .gitattributes

and manage standard and large (.bin) files without any problems

However, when I tried to make use of this cloned LFS enabled repo from inside Visual Studio by staging a new large .bin file I got an error ‘git-lfs filter-process: gif-lfs: command not found’

image

On reading around this error it suggested that the separate git-lfs package needed to be installed. I did this, making sure that the path to the git-lfs.exe (C:\Program Files\Git LFS) was in my path, but I still had the problem.

This is where I got stuck and hence needed to get some help from the Microsoft Visual Studio support team.

After a good deal tracing they spotted the problem. The path to git-lfs.exe was at the end of my rather long PATH list. It seems Visual Studio was truncating this list of paths, so as the error suggested Visual Studio could not find git-lfs.exe.

It is unclear to me whether the command prompt just did not suffer this PATH length issue, or was using a different means to resolve LFS feature. It should be noted from the command line LFS commands were available as soon as I installed Git 2.16.2. I did not have to add the Git LFS package.

So the fix was simple, move the entry for ‘C:\Program Files\Git LFS’ to the start of my PATH list and everything worked in Visual Studio.

It should be noted I really need to look at whether I need everything in my somewhat long PATH list. It’s been too long since I re-paved my laptop, there is a lot of strange bits installed.

Thanks again to the Visual Studio Support team for getting me unblocked on this.

Creating test data for my Generate Release Notes Extension for use in CI/CD process

As part of the continued improvement to my CI/CD process I needed to provide a means so that whenever I test my Generate Release Notes Task, within it’s CI/CD process, new commits and work item associations are made. This is required because the task only picks up new commits and work items since the last successful running of a given build. So if the last release of the task extension was successful then the next set of tests have no associations to go in the release notes, not exactly exercising all the code paths!

In the past I added this test data by hand, a new manual commit to the repo prior to a release; but why have a dog and bark yourself? Better to automate the process.

This can done using a PowerShell file, run inline or stored in the builds source repo and run within a VSTS build. The code is shown below, you can pass in the required parameters, but I set sensible default for my purposes

For this PowerShell code to work you do need make some security changes to allow the build agent service user to write to the Git repo. This is documented by Microsoft.

The PowerShell task to run this code is placed in a build as the only task

image

This build is then triggered as part of the release process

image

Note that the triggering of this build has to be such that it runs on a non-blocking build agent as discussed in my previous posts. In my case I trigger the build to add the extra commits and work items just before triggering the validation build on my private Azure hosted agent.

Now, there is no reason you can’t just run the PowerShell directly within the release if you wanted to. I chose to use a build so that the build could be reused between different VSTS extension CI/CD pipelines; remember I have two Generate Release Note Extensions, PowerShell and NodeJS Based.

So another step to fully automating the whole release process.

Getting ‘… is not a valid URL’ when using Git TF Clone

I have been attempting to use the Git TF technique to migrate some content between TFS servers. I needed to move a folder structure that contains spaces in folder names from a TPC that also contains spaces in its name. So I thought my command line would be

git tf clone “http://tfsserver1:8080/tfs/My Tpc” “$/My Folder”’ oldrepo --deep

But this gave the error

git-tf: “http://tfsserver1:8080/tfs/My Tpc” is not a valid URL

At first I suspected it was the quotes I was using, as I had had problems here before, but swapping from ‘ to “ made no difference.

The answer was to use the ASCII code %20 for the space, so this version of the command worked

git tf clone http://tfsserver1:8080/tfs/My%20Tpc “$/My Folder”’ oldrepo --deep

Interestingly you don’t need to use %20 for the folder name

Cloning tfs repository with git-tf gives a “a server path must be absolute”

I am currently involved in moving some TFS TFVC hosted source to a TFS Git repository.  The first step was to clone the source for a team project from TFS using the command

git tf clone --deep http://tfsserver01:8080/tfs/defaultcollection ‘$My Project’ localrepo1

and it worked fine. However the next project I tried to move had no space in the source path

git tf clone --deep http://tfsserver01:8080/tfs/defaultcollection ‘$MyProject’ localrepo2

This gave the error

git-tf: A server path must be absolute.

Turns out if the problem was the single quotes. Remove these and the command worked as expected

git tf clone --deep http://tfsserver01:8080/tfs/defaultcollection $MyProject localrepo2

Seems you should only use the quotes when there are spaces in a path name.

Updated 11 June – After a bit more thought I think I have tracked down the true cause. It is not actually the single quote, but the fact the command line had been cut and pasted from Word. This mean the quote was a ‘ not a ‘. Cutting and pasting from Word can always lead to similar problems, but it is still a strange error message, I would have expected an invalid character message

Why can’t I find my build settings on a Git based project on TFS Service?

Just wasted a bit of time trying to find the build tab on a TFS Team Project hosted on the hosted http://tfs.visualstudio.com using a Git repository. I was looking on team explorer expecting to see something like

image

But all I was seeing the the Visual Studio Git Changes option (just the top bit on the left panel above).

It took to me ages to realise that the issue was I had cloned the Git repository to my local PC using the Visual Studio Tools for Git. So I was just using just Git tools, not TFS tools. As far as Visual Studio was concerned this was just some Git repository it could have been local, GitHub, TFS Service or anything that hosts Git.

To see the full features of TFS Service you need to connect to the service using Team Explorer (the green bits), not just as a Git client (the red bits)

image

Of course if you only need Git based source code management tools, just clone the repository and use the Git tooling, where inside or outside Visual Studio. The Git repository in TFS is just a standard Git repro so all tools should work. From the server end TFS does not care what client you use, in fact it will still associate you commits, irrespective of client, with TFS work items if you use the #1234 syntax for work item IDs in your comments.

However if you are using hosted TFS from Visual Studio, it probably makes more sense to use a Team Explorer connection so all the other TFS feature light up, such as build. The best bit is that all the Git tools are still there as Visual Studio knows it is still just a Git repository. Maybe doing this will be less confusing when I come to try to use a TFS feature!

Using git tf to migrate code between TFS servers retaining history

Martin Hinshelwood did a recent post on moving source code between TFS servers using  git tf. He mentioned that you could use the –deep option to get the whole changeset check-in history.

Being fairly new to using Git, in anything other than the simplest scenarios, it took me a while to get the commands right. This is what I used in the end (using the Brian Keller VM for sample data) …

C:\tmp\git> git tf clone http://vsalm:8080/tfs/fabrikamfibercollection $/fabrikamfiber/Main oldserver –deep

Connecting to TFS…

Cloning $/fabrikamfiber/Main into C:\Tmp\git\oldserver: 100%, done.

Cloned 5 changesets. Cloned last changeset 24 as 8b00d7d

C:\tmp\git> git init newserver

Initialized empty Git repository in C:/tmp/git/newserver/.git/

C:\tmp\git> cd newserver

C:\tmp\git\newserver [master]> git pull ..\oldserver –depth=100000000

remote: Counting objects: 372, done.

remote: Compressing objects: 100% (350/350), done.

96% (358/372), 2.09 MiB | 4.14 MiB/s

Receiving objects: 100% (372/372), 2.19 MiB | 4.14 MiB/s, done.

Resolving deltas: 100% (110/110), done.

From ..\oldserver

* branch HEAD -> FETCH_HEAD

C:\tmp\git\newserver [master]> git tf configure http://vsalm:8080/tfs/fabrikamfibercollection $/fabrikamfiber/NewLocation

Configuring repository

C:\tmp\git\newserver [master]> git tf checkin –deep –autosquash

Connecting to TFS…

Checking in to $/fabrikamfiber/NewLocation: 100%, done.

Checked in 5 changesets, HEAD is changeset 30

The key was I had missed the –autosquash option on the final checkin.

Once this was run I could see my checking history, the process is quick and once you have the right command line straight forward. However, just like TFS Integration Platform time is compressed, and unlike TFS Integration Platform you also lose the ownership of the original edits.

image

This all said, another useful tool in the migration arsenal.

A bit of an edge case – Using Git-TFS to get the best (or worst?) of both worlds

Background

Whilst at the Microsoft MVP summit there are a number of MVP2MVP sessions, these are similar to DDD style sessions with MVPs presenting as opposed to Microsoft staff. One I found really interesting was one by Richard Banks based on his post on using GIT with TFS. Now this was a usage of source control tools I had not considered, a mixture of Git and TFS (or could be Git to SVN, similar tools are available)

Why do you want this usage? Especially with local workspaces coming in TFS11?

The simple answer is it allows a developer to have the advantage of Git’s multiple local versions of a given file, that they can branch, merge and rollback to as required. All prior to pushing all the changes up to a central TFS server (as opposed to GitHub or a company central Git repository).

OK lets face it this is an edge case, and it is not helped by the usage being command line driven, as opposed to be integrated with the IDE (real developers don’t use a UI or mouse, so that is OK – right?). So to try to make life a it easier I would suggest also installing Posh Git.

Setup

So what is required to get this running, if you like me a fairly new to Git there are a couple of gotcha’s. Here is the process I followed

I used Chocolaty (think Nuget for applications) to install tfsgit, this handles the dependency for the Git client

cinst tfsgit

Next I install poshgit

cinst poshgit

It is essential that you edit your Windows PATH environment variable to point to both the Git and the TFSGit folders as this is how Git picks up the extra Tfs commands, it should be something similar too this

PATH= $PATH;C:\Program Files (x86)\Git\cmd;C:\tools\gittfs

Finally for poshgit  you need runs its install script (in a PowerShell windows with elevated privileges), so it can report the number of file changes in the command prompt (note the prompt only changes when you are in a Git folder)

c:\tools\postgit\..some version\install.ps1

So hopefully this will get you going, so you can try this interesting edge case.

For more general chat on Git and distributed source control try this recent Herding Code podcast