But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Why have I got a ‘.NETCore50’ and a ‘netcore50’ folder in my nuget package?

I recently posted on how we were versioning our Nuget packages as part of a release pipeline. In test we noticed that the packages being produced by this process has an extra folder inside them.

image 

We expected there to be a netcore50 folder, but not a .NETCore50 folder. Strangely if we build the package locally we only saw the expect netcore50 folder. The addition of this folder did not appear to be causing any problem, but I did want to find out why it had appeared and remove it as it was not needed.

Turns out the issue was the version of Nuget.exe, the automatically installed version on the on-prem TFS build agent was 3.2, my local copy 3.4. As soon as I upgraded the build box’s nuget.exe version to 3.4 the problem went away

Experiences versioning related sets of NuGet packages within a VSTS build

Background

We are currently packaging up a set of UX libraries as NuGet packages to go on our internal NuGet server. The assemblies that make up the core of this framework are all in a single Visual Studio solution, however it makes sense to distribute them as a set of NuGet packages as you might not need all the parts in a given project. Hence we have a package structure as follows…

  • BM.UX.Common
  • BM.UX.Controls
  • BM.UX.Behaviours
  • etc…

There has been much thought on the versioning strategy of these packages. We did consider independent versioning of each of these fundamental packages, but decided it was worth the effort, keeping their versions in sync was reasonable  i.e. the packages have the same version number and are released as a set.

Now this might not be the case for future ‘extension’ packages, but it is an OK assumption for now, especially as it makes the development cycle quicker/easier. This framework is young and rapidly changing, there are often changes in a control that needs associated changes in the common assembly; it is hence good that a developers does not have to check-in a change on the common package before they can make an associated changed to the control package whist debugging a control prior to it being released.

However, this all meant it was important to make sure the package dependencies and versions are set correctly.

Builds

We are using Git for this project (though this process is just as relevant for TFVC) with a development branch and a master branch. Each branch has its own CI triggered build

  • Development branch build …
    • Builds the solution
    • Runs Unit tests
    • Does SonarQube analysis
    • DOES NOT store any built artifacts
    • [Is used to validate Pull requests]
  • Master branch build …
    • Versions the code
    • Builds the solution
    • Runs Unit tests
    • Creates the NuGet Packages
    • Stores the created packages (to be picked up by a Release pipeline for publishing to our internal NuGet server)

Versioning

So within the Master build we need to do some versioning, this needs to be done to different files to make sure the assemblies and the NuGet packages are ‘stamped’ with the build version.

We get this version for the build number variable, $(Build.BuildNumber), we use the format $(Major).$(Minor).$(Year:yy)$(DayOfYear).$(rev:r)  e.g. 1.2.16123.3

Where

  • $(Major) and $(Minor) build variables we manage (actually our release pipeline updates the $(Minor) on every successful release to production using a VSTS task)
  • $(Year:yy)$(DayOfYear) gives a date in the form 16123
  • and $(rev:r) is a count of builds on a given day

We have chosen to use this number format to version both the assemblies and Nuget packages, if you have different plans, such as semantic versioning , you will need to modify this process a bit.

Assemblies

The assemblies themselves are easy to version, we just need to set the correct value in their assemblyinfo.cs or assemblyinfo.vb files. I used my Assembly versioning VSTS task to do this

NuGet Packages

The packages turn out to be a bit more complex. Using the standard NuGet Packager task there is a checkbox to say to use the build number as the version. This works just fine versioning the actual package, adding the –Version flag to the package command to override the value in the project .nuspec file. However it does not help with managing the versions of any dependant packages in the solution, and here is why. In our build …

  1. AssemblyInfo files updated
  2. The solution is built, so we have version stamped DLLs
  3. We package the first ‘common’ Nuget package (which has no dependencies on other projects in the solution) and it is versioned using the –version setting, not the value in it’s nuspec file.
  4. We package the ‘next’ Nuget package, the package picks up the version from the –version flag (as needed), but it also needs to add a dependency to a specific version of the ‘common’ package. We pass the –IncludeReferencedProjects  argument to make sure this occurs. However, Nuget.exe gets this version number from  the ‘common’ packages .nuspec file NOT the package actually built in the previous step. So we end up with a mismatch.

The bottom line is we need to manage the version number in the .nuspec file of each package. So more custom VSTS extensions are needed.

Initially I reused my Update XML file task, passing in some XPath to select the node to update, and this is a very valid approach if using semantic versioning as it is a very flexible way yo build the version number. However, in the end I added an extra task to my versioning VSTS extension for Nuget to make my build neater and consistent with my other versions steps.

Once all the versioning was done I could create the packages. I ended up with a build process as shown below

image

A few notes about the NuGet packaging

  • Each project I wish to create a Nuget package for has a nuspec file of the same ‘root’ name in the same folder as the csproj eg. mypackage.csproj and mypackage.nuspec. This file contains all descriptions, copyright details etc.
  • I am building each package explicitly, I could use wildcards in the ‘Path/Pattern to nuspec files’ property, I choose not to at this time. This is down to the fact I don’t want to build all the solution’s package at this point in time.
  • IMPORTANT I am passing in the .csproj file names, not the .nuspec file names to the ‘Path/Pattern to nuspec files’ property. I found I had to do this else the   –IncludeReferencedProjects  was ignored. The Nuget documentation seems to suggest as long as the .csproj and .nuspec files have the same ‘root’ name then you could reference the .nuspec file but this was not my experience
  • I still set the flag to use the build version to version the package – this is not actually needed as the .nuspec file has already been update
  • I pass in the  –IncludeReferencedProjects  argument via the advanced parameters, to pick up the project dependancies.

Summary

So now I have a reliable way to make sure my NuGet packages have consistent version numbers 

Video on Nuget for C++ on Channel 9

I have been out to a number of sites recently where there are C++ developers. We often get talking about package management and general best practices for shared libraries. The common refrain is ‘I wish we had something like Nuget for C++’.

Well it was released in Nuget 2.5 and there is a video on Channel9 all about it.

403 and 413 errors when publishing to a local Nuget Server

We have an internal Nuget Server we use to manage our software packages. As part of our upgrade to TFS2012 this needed to be moved to a new server VM and I took the chance to upgrade it from 1.7 to 2.1.

The problem

Now we had had a problem that we could publish to the server via a file copy to its underlying Packages folder (a UNC share) but could never publish using the Nuget command e.g.

Nuget push mypackage.nupkg -s http://mynugetserver

I had never had the time to get around to sorting this out until now.

The reported error if I used the URL above was

Failed to process request. 'Access denied for package 'Mypackage.'.
The remote server returned an error: (403) Forbidden..

If I changed the URL to

Nuget push mypackage.nupkg -s http://mynugetserver/nuget

The error became

Failed to process request. 'Request Entity Too Large'.
The remote server returned an error: (413) Request Entity Too Large..

Important: This second error was a red herring, you don't need the /nuget on the end of the URL

The solution

The solution was actually simple, and in the documentation though it took me a while to find.

I had not specificed an APIKey in the web.config on my server, obvious really my access was blocked as I did not have the shared key. The 413 errors just caused me to waste loads of time looking at WCF packet sizes because I had convinced myself I needed to use the same URL as you enter in Visual Studio > Tools > Option > Package Management > Add Source, which you don't

Once I had edited my web.config file to add the key (or I could have switched off the requirement as an alternative solution)

  <appSettings>
    <!--
            Determines if an Api Key is required to push\delete packages from the server.
    -->
    <add key="requireApiKey" value="true" />
    <!--
            Set the value here to allow people to push/delete packages from the server.
            NOTE: This is a shared key (password) for all users.
    -->
    <add key="apiKey" value="myapikey" />
    <!--
            Change the path to the packages folder. Default is ~/Packages.
            This can be a virtual or physical path.
        -->
    <add key="packagesPath" value="" />
  </appSettings>

I could then publish using

Nuget mypackage.nupkg myapikey -s http://mynugetserver

Using an internal Nuget server to manage the Typemock assembly references.

In my last post I discussed the process I needed to go through to get Typemock Isolator running under TFS 2012. In this process I used the Auto Deploy feature of Isolator. However this raised the  question of how to manage the references within projects. You cannot just assume the Typemock assemblies are in the GAC, they are not on the build box using auto deploy. You could get all projects to reference the auto deployment location in source control. However, if you use build process templates across projects it might be you do not want to have production code referencing build tools in the build process are directly.

For most issues of this nature we now use Nuget. At Black Marble we make use of the public Nuget repository for tools such as XUnit, SpecFlow etc. but we also have an internal Nuget repository for our own cross project code libraries. This includes licensing modules, utility and data loggers etc.

It struck me after writing the last post that the best way to manage my Typemock references was with a Nuget package, obviously not a public one, this would be for Typemock to produce. So I create one to place on our internal Nuget server that just contained the two DLLs I needed to reference (I could include more but we usually only need the core and act assert arrange assemblies).

[Update 6th Aug PM] – After playing with this today seems I need the following in my Nuget package

Lib
    Net20
         Configuration.dll
         Typemock.ArrangeActAssert.dll
         TypeMock.dlll

If you miss out the configuration.dll it all works locally on a developers PC, but you get a ‘cannot load assembly error’ when trying to run a TFS build with Typemock auto deployment. Can’t see why obviously but adding the reference (assembly to the package) is a quick fix.

 

image

IT IS IMPORANT TO NOTE that using a Nuget package here in no way alters the Typemock licensing. Your developers still each need a license, they also need to install Typemock Isolator, to be able to run the tests and your build box needs to use auto deployment. All using Nuget means is that you are now managing references in the same way for Typemock as any other Nuget managed set of assemblies. You are internally consistent, which I like.

So in theory as new versions of Typemock are released I can update my internal Nuget package allowing projects to use the version they require. It will be interesting to see how well this works in practice.

Using Nuget and TFS Build 2010

At one of our recent events I was asked if I had any experience using Nuget within a TFS 2010 build. At the time I had not, but I thought it worth a look.

For those of you who don’t know Nuget is a package manager that provides a developer with a way to manage assembly references in a project for assemblies that are not within their solution. It is most commonly used to manage external commonly used assemblies such a nHibernate or JQuery but you can also use it manage your own internal shared libraries.

The issue the questioner had was that they had added references via Nuget to a project

image

Their project then contained a packages.config file that listed the Nuget dependencies. This was in the project root with the <project>.csproj file.

<?xml version="1.0" encoding="utf-8"?>
<packages>   <package id="Iesi.Collections" version="3.2.0.4000" />   <package id="NHibernate" version="3.2.0.4000" />
</packages>

This packages.config  is part of the Visual Studio project and so when the project was put under source control so was it.

However, when they created a TFS build to build this solution all seems OK until the build ran, when they got a build error along the lines

Form1.cs (16): The type or namespace name 'NHibernate' could not be found (are you missing a using directive or an assembly reference?)
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets (1490): Could not resolve this reference. Could not locate the assembly "Iesi.Collections". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets (1490): Could not resolve this reference. Could not locate the assembly "NHibernate". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.

Basically the solution builds locally but not on the build box, the assemblies referenced by Nuget are missing. A quick look at the directory structure show why. Nuget stores the assemblies it references in the solution folder, so you end up with

Solution Directory
      Packages – the root of the local cache of assemblies created by Nuget
      Project Directory

If you look in the <project>.csproj  file you will see a hint path pointing back up to this folder structure so that the project builds locally

<Reference Include="NHibernate">
      <HintPath>..\packages\NHibernate.3.2.0.4000\lib\Net35\NHibernate.dll</HintPath>
</Reference>

The problem is that this folder structure is not known to the solution (just to Nuget), so this means when you add the solution to source control this structure is not added, hence the files are not there for the build box to use.

To fix this issue there are two options

  1. Add the folder to source control manually
  2. Make the build process aware of Nuget and allow it to get the files it needs as required.

For now lets just use the first option, which I like as in general in do want to build my projects against a known version of standard assemblies, so putting the assemblies under source control is not an issue for me. It allows me to easily go back to the specific build if I have to.

(A quick search with your search engine of choice will help with the second option, basically using the nuget.exe command line is the core of the solution)

To add the files to source control, I when into Visual Studio > Team Explorer > Source Control and navigated to the correct folder. I then pressed the add files button and added the whole Packages folder. This is where I think my questioner might have gone wrong. When you add the whole folder structure the default is to exclude .DLLs (and .EXEs)

image

If you don’t specifically add these files you will still get the missing references on the build, but could easily be thinking ‘ but I just added them!’, easy mistake to made, I know I did it.

Once ALL the correct files are under source control the build works as expected.