But it works on my PC!

The random thoughts of Richard Fennell on technology and software development

Unit testing in VS11Beta and getting your tests to run on the new TFSPreview build service

One of my favourite new features in VS11 is that the unit testing is pluggable. You don’t have to use MSTest, you can use any test framework that an adaptor is available for (at the release of the beta this meant the list of framworks on Peter Provost’s blog, but I am sure this will grow).

So what does this mean and how do you use it?

Add some tests

First it is worth noting that you no longer need to use a test project to contain your MSTest, you can if you want, but you don’t need to. So you can

  1. Add a new class library to your solution
  2. Add a reference to Microsoft.VisualStudio.TestTools.UnitTesting and create an MStest test
  3. Add a reference to xUnit (I used NuGet to add the reference) and create an XUnit test
  4. Add a reference to XUnit extensions (NuGet again) and add a row based xUnit test
  5. Add a reference to nUnit (you guessed it - via NuGet) and create a nUnit test

All these test frameworks can live in the same assembly.

Add extra frameworks to the test runner

By default the VS11 test runner will only run the MStest test, but by installing the xUnit.net runner for Visual Studio 11 Beta and NUnit Test Adapter (Beta) either from the Visual Studio gallery or via the Tools –> Extension Manager (and restarting VS) you can see all the test are run

image

You can if you want set it so that every time you compile the test runner triggers (Unit Testing –> Unit Test Settings –> Run Test After Build). All very nice.

image

Running the tests in an automated build

However, what happens when you want to run these tests as part of your automated build?

The build box needs to have have a reference to the extensions. This can be done in three ways. However if you are using the new TFSPreview hosted build services, as announced at VS Live, only one method, the third, is open to you as you have not access to the VM running the build to upload files other than by source control.

By default, if you create a build and run it on the hosted build you will see it all compiles, but only the MStest test is run

image

The fix is actually simple.

  1. First you need to download the xUnit.net runner for Visual Studio 11 Beta and NUnit Test Adapter (Beta) .VSIX packages from Visual Studio Gallery.
  2. Rename the downloaded files as a .ZIP file and unpack them
  3. In TFSPreview source control create a folder under the BuildProcessTemplates for your team project. I called mine CustomActivities (the same folder can be used for custom build extensions hence the name, see Custom Build Extensions for more details)
  4. Copy the .DLLs from the renamed .VSIX files into this folder and check them in. You should have a list as below

    image
  5. In the Team Explorer –> Build hub, select the Actions menu option –> Manage Build Controllers. Set the Version control path for  custom assemblies to the new folder.

    image

You do not need to add any extra files to enable xUnit or nUnit tests as long as you checked in the runtime xUnit and nUnit assemblies from the Nuget package at the solution level. This should have been default behaviour with NuGet in VS11 (i.e. there should be a package folder structure in source control as shown in source explorer graphic above)

You can now queue a build and you should see all the tests are run (in my case MStest, XUnit and nUnit). The only difference from a local run is that the xUnit row based tests appear as separate lines in the report

image

So now you can run tests for any type on a standard TFSPreview hosted build box, a great solution for many projects where just a build and test is all that is required.

Problems editing TFS11 build templates in VS11Beta

Whilst writing documentation for TFS community build extensions (just published the Zip activity documentation) I hit upon a problem working with TFS11. The TFS community build extensions support both TFS2010 and TFS11beta, unfortunately the two versions need to be built separately (once against TFS2010 DLLs and once against TFS11 ones). As of version 1.3 of the extensions both versions are shipped in the download.

In the past in the past I have tended to work in TFS2010 on this community project, but since the VS/TFS11 beta release I am trying to move over to the new build. So to write the documentation for the ZIP activity I started in TFS11. I followed the usual method to use a custom activity (there is no improvement over this frankly horrible process in VS11) so within VS11 I added the ZIP activity to a copy of the defaultprocesstemplate.xaml. All appeared OK but when I ran a build with this new template. I got the error

The build process failed validation. Details:

Validation Error: The private implementation of activity '1: DynamicActivity' has the following validation error:   Compiler error(s) encountered processing expression "BuildDetail.BuildNumber".

Type 'IBuildDetail' is not defined.

image

On checking the .XAML file you can see there is duplication in the namespaces

image

[Note the greatly improved compare tooling in VS11]

This it  turns out is a known issue logged on Microsoft Connect. The answer, at this time, is to do you build process .XAML editing in a text editor like Notepad, not a good story, but a workaround.

More thoughts on Typemock Isolator, Microsoft Fakes and Sharepoint

I posted yesterday on using Typemock and Microsoft Fakes with SharePoint. After a bit more thought I realised the key thing in using Typemock I found easier was the construction of my SPListItem dataset. Typemock allowed me to fake SPListItems and put them in a generic List<SPListItem> then just make this the return value for the Item collection using the magic .WillReturnCollectionValuesOf() method that converts my List to the required collection type. With the Microsoft Fakes I had think about a delegate that constructed my test data at runtime. This is not a problem, just a different way of working.

A side effect of using the Typemock .WillReturnCollectionValuesOf() method is that if I check the number of SPListItems in the return SPListColection I have a real collection so I can use the collection’s own .Count method, I don’t have fake it out. With the Microsoft Fakes as there is no collection returned so I must Fake its return value.

This is a trend common across Typemock Isolator, it does much  of the work for you. Microsoft Fakes, like Moles, required you to do the work. In Moles this was addressed by the use of behaviour packs to get you started with standard items you need in SharePoint.

I would say again that there may be other ways of using the Microsoft Fakes library, so there maybe ways to address these initial comments of mine, I am keen to see if this is the case

Now that VS11 has a fake library do I still need Typemock Isolator to fake out SharePoint?

Updated 5 May 2012 Also see my follow up post, this corrects some of the information of faking SharePoint 

I have done posts in the past about how you can use Typemock Isolator to fake out SharePoint to speed design and testing. The  reason you need special tooling, beyond standard mocking frameworks like Rhino or MOQ, is that SharePoint has many sealed private classes with no public constructors. So in the past you only had two options: Typemock Isolator and Moles from Microsoft research.

With the release of the Visual Studio 11 beta we now have a means to fake out ‘non mockable classes’ (shim) classes that is shipped in the box. This tooling I understand has it roots in Moles, but is all new. So with the advent of fakes in VS11 you have to ask ‘do I still need Typemock isolator?”

To answer this question I have tried to perform the same basic mocking exercise as I did in my previous posts. Create a fake SharePoint list and make sure I can access the faked out content in test asserts.

In Typemock Isolator

To perform the test in Isolator, assuming Isolated is installed on your PC, is add a reference to Typemock.dll and Typemock.ArrangeActAssert.dll and use the following code

[TestMethod]
        public void FakeSharePointWithIsolator()
        {

            // Arrange // build the dataset var fakeItemList = new List<SPListItem>();
            for (int i = 0; i < 3; i++)
            {
                var fakeItem = Isolate.Fake.Instance<SPListItem>();
                Isolate.WhenCalled(() => fakeItem.Title).WillReturn(String.Format("The Title {0}", i));
                Isolate.WhenCalled(() => fakeItem["Email"]).WillReturn(String.Format("email{0}@fake.url", i));

                fakeItemList.Add(fakeItem);
            }

            // fake the SPWeb and attach the data var fakeWeb = Isolate.Fake.Instance<SPWeb>();
            Isolate.WhenCalled(() => fakeWeb.Url).WillReturn("http://fake.url");
            Isolate.WhenCalled(() => fakeWeb.Lists["fakelistname"].Items).WillReturnCollectionValuesOf(fakeItemList);


            // act // not actually doing an operation // assert Assert.AreEqual("http://fake.url", fakeWeb.Url);
            Assert.AreEqual(3, fakeWeb.Lists["fakelistname"].Items.Count); 
            Assert.AreEqual("The Title 0", fakeWeb.Lists["fakelistname"].Items[0].Title);
            Assert.AreEqual("email0@fake.url", fakeWeb.Lists["fakelistname"].Items[0]["Email"]);
            Assert.AreEqual("The Title 1", fakeWeb.Lists["fakelistname"].Items[1].Title);
            Assert.AreEqual("email1@fake.url", fakeWeb.Lists["fakelistname"].Items[1]["Email"]);
            
        }

Using Microsoft Faking

Adding the Fake

The process to added a fake in VS11 is to right click on an assembly reference (in our case Microsoft.SharePoint) and select the ‘add fake assembly’ option.

image

You should see a Fake reference created and an entry in the fakes folder

image

Gotcha Warning ‘Can’t generate the fake reference’ – When I tried to generate a fake for the Microsoft.SharePoint assembly within a classlibrary project that had a reference to only the Microsoft.Sharepoint assembly (and the default assemblies references added for any classlibrary project) the entry is made in the Fakes folder but no .Fakes assembly is created. After a delay (30 seconds?)  you see an error message in the Visual Studio output window. This tells you a reference cannot be resolved, if you delete the entry in the Fakes folder, add the missing reference listed in the output windows and repeat the process you get the same problem but another assembly is named as missing, add this and repeat this process. Eventually the .Fakes assembly is created.

In the case of this SharePoint sample I had to manually add Microsoft.SharePoint.Dsp, Microsoft.SharePoint.Library, Microsoft.SharePoint.Search, System.Web, System.Web.ApplicationServices. Remember these entries are ONLY required to allow the fake creation/registration, they are not needed for your assembly to work in production or for Typemock.  [5 May 2012 See my follow up post

You should now have a generated assembly that you can use to create your fakes shims

Writing the fakes logic

The logic to create the fake behaviour is as follows. Now there might be easier ways to do this, but this does work and is reasonably readable

 [TestMethod] public void FakeSharePointWithShims() 
{
using (ShimsContext.Create()) // required to tidy up the shim system
{
// arrange
var fakeWebShim = new Microsoft.SharePoint.Fakes.ShimSPWeb()
{
UrlGet = () =>
http://fake.url,
ListsGet = () =>
new Microsoft.SharePoint.Fakes.ShimSPListCollection()
{
ItemGetString = (listname) =>
new Microsoft.SharePoint.Fakes.ShimSPList()
{
ItemsGet = () =>
new Microsoft.SharePoint.Fakes.ShimSPListItemCollection()
{
// we have to fake the count, as we not returning a list SPListCollection
CountGet = () => 3,
ItemGetInt32 = (index) =>
new Microsoft.SharePoint.Fakes.ShimSPListItem()
{
TitleGet = () =>
string.Format("The Title {0}", index),
ItemGetString = (fieldname) =>
string.Format("email{0}@fake.url", index)
// note we don't check the field name
}
}
}
}
};

// act
// not actually doing an operation
// assert
var fakeWeb = fakeWebShim.Instance;
Assert.AreEqual("http://fake.url", fakeWeb.Url);
Assert.AreEqual(3, fakeWeb.Lists["fakelistname"].Items.Count);
Assert.AreEqual("The Title 0", fakeWeb.Lists["fakelistname"].Items[0].Title);
Assert.AreEqual("email0@fake.url", fakeWeb.Lists["fakelistname"].Items[0]["Email"]);
Assert.AreEqual("The Title 1", fakeWeb.Lists["fakelistname"].Items[1].Title);
Assert.AreEqual("email1@fake.url", fakeWeb.Lists["fakelistname"].Items[1]["Email"]); }

 

Comments

Which which do you find the most readable?

I guess it is down to familiarity really. You can see I end up using the same test asserts, so the logic I am testing is the same.

I do think the Typemock remains the easier to use. There is the gotcha I found with having to add extra references to allow the fakes to be create in the VS11 faking system, and just the simple fact of having to manually create the fakes in Visual Studio, as opposed to it just being handled behind the scenes by Typemock. There is also the issue of having to refer to the ShimSPWeb class and using the .Instance property as opposed to just using the SPWeb in Typemock.

The down side of Typemock is the cost and the need to have it installed on any development and build machines, neither an issue for the Microsoft fake system, the tests just being standard .NET code that can be wrappered in any unit testing fame work (and remember VS11’s Unit Test Explorer and TFS11 build allow you to use any testing framework not just MStest.

So which tool am I going to use? I think for now I will be staying with Typemock, but the VS11 faking system is well worth keeping an eye on.

[Updates 24 March 12 - More comments added in a second post]

Typemock Isolator Version 7

Whilst travelling I had neglected to post about the new release of Typemock Isolator Version 7. This new release extends the range of the product to provide a variety of ‘test support tools’ that help you track down bugs sooner.

However, the best new feature for me is that it allows support of historic versions of Isolator, this means you don’t have to upgrade all projects to V7 at the same time. The V7 engine will use the older assemblies  e.g. V6 perfectly happily. Make managing build boxes far easier

Dropping build output to source control in TFS11

One of the nice new feature of TFS11 is that you get a third option for what to do with your build output

  • Don’t copy output anywhere – good for CI builds that nobody will ever consume, just used to run tests
  • Drop to a UNC share e.g. \\server1\drops – the default and used 9 times out 10
  • The new one - drop to source control e.g. $/myproject/drops.

The advantage of this new third option is your build agents can place the files they create in a location that can be accessed by any TFS client i.e. in the source control repository. A user no longer needs to be on a VPN or corporate LAN to be able to see a UNC share.

But remember, just because the builds are in source control does not mean that the build don’t still follow the normal build retention policies, so they will not accumulate forever, unless you want them to.

Now some teams will have good reasons as to why the don’t want builds going into source control. Deployments to a NuGet server and the like will be a far better system for them. This is still all possible, it is just down to build process template customisation. You have not lost any options, just gained another one out the box

But what about building Java via Ant or Maven within TFS using the Build Extensions? Well at this time the process template used to create this type of build from within Eclipse has not caught up with this new feature. However if you really want it you can do the following

  1. Create a TFS build in Eclipse that drops to a UNC share
  2. Open the build definition in VS11
  3. Edit the drops location to point to a location in source control and save the build

    image
  4. When you trigger a new build and you should get you drops in source control. Note in the confirmation dialog you can see the source control based path but you can’t edit it (if you try you get an invalid path error)

    image

Team Explorer Everywhere is now free

It was announced overnight that TEE is now free. What does this mean?

It means if you do not have to buy TEE as some extra software if you already have a TFS CAL. This removed a barrier to adoption for developers who work in heterogeneous systems, there is no extra cost to integrate Eclipse as well a Visual Studio with TFS .

If you want to find out more about TEE why not come Black Marble’s free webinar I am delivering on the 19th?