GUI Testing Framework – First software release

1 INTRODUCTION

1.1 The Problem

The graphical user interface (GUI) of a computer system is one of the most troublesome sub-systems to test. Though some products have been marketed to address this area they have not been that well-received or effective. All such products have tended to be based on macro recording systems that often introduce more errors than originally present in the software they aim to test. The net effect of this is that the test engineer spends more time debugging their test scripts than they do testing their product, as the slightest change in a GUI layout will usually break the test harness. Hence up to now the most effective way to test a GUI is to get human testers to do it, though this is not without problems of its own.

However these problems can be avoided when testing non-GUI components; a variety of excellent tools and techniques exist to address this area such as JUnit and NUnit as well as commercial products from companies. As in many applications the GUI can be separated from the business logic, these tools can be used to test many features of a system. However, the problem remains that these products are not directly applicable to the testing of GUIs.

1.2 A Potential Solution

The lack of GUI testing is a major, but currently unavoidable, hole in current software engineering processes. Only so much can be done when separating the GUI from unit testable backend functions.

This sample shows how this problem can be tackled with the Microsoft .NET framework. .NET provides two technologies that help address this problem:

  • Attributes – The ability for the developer to add custom metadata to their program assembly that can provide information on their intentions and the use of the assembly for both static analysis and at run time operations.
  • Reflection – The ability of a .NET program to read the metadata of an assembly and use it to create objects and execute processes as required.

The basic method is to provide metadata on Windows controls and events such that a test harness can execute the Windows application and ‘watch for’ an appropriate response e.g. when button X is pressed then window Y should appear.

2 TOOLS USED

2.1 Microsoft Visual Studio 2003 and C#

All code is written in the Visual Studio 2003 version of C#. This requires the .NET framework 1.1.

2.2 FXCop

FXCop  is a free tool for Visual Studio 2003 provided by Microsoft, and built into Visual Studio 2005. It provides a means to automatically check that .NET code is written to current recommended design and style guidelines. It can be extended with a developers own rules if required.

In this project the default rule set has been used.

2.3 Automatic XML Documentation

All public fields, properties and methods within the project are documented using the C# XML documentation system.

This XML documentation is used to build a documentation web site using built in tools within Visual Studio and a standard Windows help file using nDOC .

2.4 Source Control

All software source code was stored under Microsoft Visual Source Safe revision control software during development.

The code provided in the sample package should have had the links to VSS removed.

3 USAGE OF THE TEST HARNESS

The basic tools for the system are provided in the GUITesterSampleV1.zip file. The full source code is provided in GUITesterSource-V1.zip, and the user manual in GUITester Usage Manual.pdf

3.1 Marking Up Application for Testing

Attributes are applied to an application in a similar manner to nUnit.

A fully marked up sample application has been provided.

3.1.1 Marking a Class as Testable

To mark a class as testable as single attribute needs to be applied to the class definition.
 [GuiTestable]
 public class Form1 : System.Windows.Forms.Form {…}

Once this is added the class should appear in tester application when the assembly is loaded (see section 3.2).

3.1.2 Defining Tests

The following test types are available

3.1.2.1 ClickDataGridCountTest

This test type can be applied to any GUI control that issues a click event. It counts the number of rows in a DataGrid control.

It is applied to the definition of the GUI control as shown below:

  [ClickDataGridCountTest(“Click DG Test 1″,”dg”,5)]
  private System.Windows.Forms.Button myButton;

or

  [ClickDataGridCountTest(“Click DG Test 2″,”dg”,0,5)]
  private System.Windows.Forms.Button myButton;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The name of the target DataGrid
  • The initial value to expect before the click event (optional)
  • The final value to expect after the click event

3.1.2.2 ClickOpenFormTest

This test type can be applied to any GUI control that issues a click event. It checks to see if a suitably named window is opened then the click event occurs.

It is applied to the definition of the GUI control as shown below:

  [ClickOpenFormTest(“Open Window Test”,”Form2″,true)]
  private System.Windows.Forms.Button myButton;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The Window name of the forms that should be searched for.
  • Set to true of the Window should be closed once it has been detected

3.1.2.3 ClickOpenMsgBoxTest

This test type can be applied to any GUI control that issues a click event. It checks to see if a standard MessageBox has been displayed with the expected message.

  [ClickOpenMsgBoxTestAttribute(“Click Msg Test”,”Message”)]
 private System.Windows.Forms.MenuItem myMenu;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The message text expected

3.1.2.4 ClickTextTest

This test type can be applied to any GUI control that issues a click event. It checks the contents of any GUI control that supports the .Text property.

It is applied to the definition of the GUI control as shown below:

[ClickTextTest(“Click 1″,”textBox”,”Hello World”)]
  private System.Windows.Forms.MenuItem myMenuItem;

or

  [ClickTextTest(“Click 2″,”label”,”Before”,”After”)]
private System.Windows.Forms.Button myButton;

The parameters are:

The name of the test (for the benefit of the developer, not used by the testers in any way)

  • The name of the target GUI Control that contains the text
  • The initial value to expect before the click event (Optional)
  • The final value to expect after the click event

3.1.2.5 TextSizeTest

This test type can be applied to any GUI control that contains text in a .Text field. It checks that the text currently displayed is fully visible.

It is applied to the definition of the GUI control as shown below:

  [TextSizeTest(“Size test”)]
 private System.Windows.Forms.TextBox textBox;

The parameter is:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)

3.1.2.6 UserDefinedTest

Unlike all other tests, this type of test is applied to a class, not a GUI control.

[UserDefinedTest(“User Test”,”UserTest1″)]
 public class Form1 : System.Windows.Forms.Form {…}

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The name of the method to run as the test
  • The test method should be in the form as shown below, returning true if the test is passed.

This method can contain any valid .NET code.
///

  /// User defined test
  ///

  /// True if passed
  public bool UserTest1()
  {
   return false;
 }

3.2 Graphical Testing Tool

Load the GUITestHarness.exe (found in the GUITesterTestHarnessbinDebug directory). This tools follows the design model of the tester in nUnit.

Use the load option on the file menu to open the sample application for testing. You should see the available tests in the tree control.
 
The tests can be run by pressing the ‘Run Test’ button or by using the Tests menu. In both cases an option is provided to run one or all of the tests.

3.3 Command Line Testing  Tool

The command line tool is run in the following way from a DOS box

ConsoleTestHarness.exe MyAssembly.DLL
 
The only differences from the GUI tester are:

  • That the command line creates a new instance of the object under test for each test in turn. This may need to be reviewed in the future.
  • All tests are run; there is no way to control the running of a subset.

The command line tool provides a simple output to the StdOut (the command line) but also returns a DOS error level for success for failure.

4 REVIEW OF THE PROJECT

4.1 Is the result worth the effort?

The measure of any testing system is whether the effort in setting up the tests and maintaining them outweighs the cost in doing manual tests. Most testing products fail this test; this is especially true for GUI testing. The key problem with GUI testing is that most tools rely on knowing where the controls under test are on the page. The testing model in this project removes this requirement. The management of tests is based on the logical structure of the Windows form, not some XY grid locations.

Given this fact this testing model is a success. Once the developer has added a test, hopefully, if using a ‘test driven development model’ before they implement the actual application code, then the test should remain valid for the lifetime of the control it tests. This should be true irrespective of whether the control is moved, resized or any of its display properties changed.

4.2 Is the system dynamic enough?

However, this does not mean that this project offers an outright solution to all GUI testing. In essence it is a unit testing model, and all unit testing models have the issue that they test blocks of code statically. Usually a special instance of the class under test is created and the tests run. This means that often unit testing tends to be limited to testing libraries of functions that are easy to handle statically e.g. finance operations, if I have these parameters do I get the correct numeric answer?

This becomes a problem for GUI testing models, no GUI is static; its job is to dynamically display information generated by backend business logic. Care has to be taken when designing tests that the Windows form does have the required underlying business features, such as connections to databases, to allow meaningful tests.

4.3 Is it usable for a developer?

Given all these facts this is a viable testing system, but whether it would be taken up by the .NET developer community is another question.

The NUnit project, since the inception of this project, has released a system called NUnitForm . This tries to address similar issues to those addressed in this project, but building on the NMock  system to create stub instances of any business and GUI classes required.

Due to this architecture, NUnitForm becomes a very comprehensive and flexible system, but fairly complex to use. The system defined in this project is far simpler to use, but may be too inflexible for practical use because of this simplicity. However this project’s relative simplicity may be its greatest advantage.

Also any product based on NUnit has a good degree of momentum within the development community.

4.4 What are the current limitations?

In its current form this framework has some significant limitations. However these are not fundamental, they are just issues that have yet to be fully resolved. With a suitable application of time and effort they can be addressed.

4.4.1 Object Hierarchies

The test harness should be able to explorer the hierarchy of classes within an assembly. This requires that the test discovery process should use a tree walker model as opposed to a simple list model.

4.4.2 MDI Forms

Not all Windows applications are SDI, many use a MDI format. The test harness should be able to test MDI applications. This is related to the issues raised in 4.4.1.

In MDI applications there are two types of form: an MDI parent (the container) and MDI children (the forms within the container), both should be testable.

In most cases testing of the individual MDI child forms, running as temporary SDI forms will suffice. However this does not allow for testing of inter-form MDI form interactions. To do this further work will need to be undertaken using the Windows API to explore the inner contents of an MDI application using its various Window handles

A mechanism for testing the interrelation of child MDI forms would greatly add to the possible uses for the system.

4.4.3 Application object

It was noted that use of the Application object was not possible in the test harness. One of the side effects of this problem is that the standard means for providing user configuration information is not available in the test environment. The standard .NET means to provide configuration information is to use an XML file, by default if a program is called MYAPP.EXE the configuration file is called MYAPP.EXE.CONFIG. This is looked for in the same directory as the .EXE.
A partial solution has been provided by the use of a wrapper class when trying to use resources in the directory that contains the assembly under test.

If a full solution can be found to the discovery of the path to the assembly under test, then the testing framework becomes far easier to integrate with existing applications as fewer changes in the application code will be required.

It is thought that this is the one limitation that will be the hardest to resolve.

4.4.4 Test Order & Setup

The lack of a flexible means to order tests is a major limitation at this time. The knock-on effects of one test can cause another to fail when it should not. This can be resolved by the addition of means to define setup and rip down methods, and a more formal way to order tests or, even better, a means to sub group them.

4.4.5 Threading

The tests with MessageBox’s showed that for any modal Windows operations a multi-thread model was unavoidable.  Migrating the testing tools to a full multi-thread model should be considered a priority. Currently multi-threading is only used for the MessageBox test. However a developer using this framework should not have to worry whether they application is modal or not. The test harness should be able to handle the form irrespective of modal issues.

5 POSSIBLE FURTHER ENHANCEMENTS

The current list of areas for possible further enhancements is:

  • Counting the number of items in a ListBox, probably similar to the DataGrid test.
  • Radio buttons click – need to check how having the radio button in a group (so only one radio button can be checked at any one time) effect the tests
  • Do group boxes and panels within a form effect how controls are found? It is assumed that the use of the GetValue call may have removed this as an issue
  • Initial state and test order. Do we need a set of [setup] attributes (ones to run once for a test run and ones to run before each individual test) to prepare the application for testing?
  • Do we need to force a test order?
  • It is not possible to load a running .EXE into the test harness? Currently, due to the way assemblies are loaded, they are locked by the operating system. Is there a way round this problem?
  • How best can the MDI children, inside an MDI container, be tested?
  • Can the set of tests be nested within the test harness using the sample applications class hierarchy?
  • Make the testing system multi-threaded


This work is copyright © Black Marble Ltd. 

Indigo and Avalon Beta1 (RC)

Indigo and Avalon Beta1 (RC) has been posted at Avalon and Indigo Beta 1 RC

This beta targets Visual Studio 2005 Beta2 and the only words I can use to describe it is totally awesome. We migrated all our Visual Studio 2005 installations to Beta2 at release due to the dependences for Avalon/Indigo/DSL tools we will not be implementing any remaining CTP releases, stroll on the Beta2 compatible DSL toolkit.

We will be producing some demonstrations for Indigo and Avalon during the summer so watch this space. If there is enough interest we may ( Microsoft Licensing permitting ) do an end of day session on Avalon and Indigo during our introduction to Visual Studio and SQL Server 2005 event in October.

Robert

Web Enabling your Business

Black Marble’s latest seminar, Web Enabling your Business, takes place next Wednesday (18 May) at Touchbase. 

Come for the usual high standard of content … and stay for the food!

If you want to reserve a place, check out our website for details.

Richard’s ‘automated GUI testing in .NET’ project

Testing of software, and specifically the automated testing of software, is an area I have always been interested in. Recently I have been working on ways of automating the testing of .NET Windows Form applications.

Today I have posted an article The automated testing of graphical user interfaces in .NET as a taster as to what I have been doing, I will be posting a more articles with sample test harnesses and an extensible API soon.

Richard.

The automated testing of graphical user interfaces in .NET

The Problem
The graphical user interface of a computer system is one of the most troublesome sub-systems to test. Though some products have been marketed to address this area they have not been that well-received or effective. All such products have tended to be based on macro recording systems that are awkward to implement and keep up to date. The net effect of this is that the test engineer spends more time debugging their test scripts than they do testing their product, as the slightest change in a GUI layout will usually break the test harness. Hence up to now the most effective way to test a GUI is to get human testers to do it, though this is not without problems of its own, humans make mistakes, especially when doing boring repetitive tasks.

The Potential Solution
The lack of GUI testing is therefore a major, but currently unavoidable, hole in current software engineering processes. Only so much can be done when separating the GUI from the business logic and using unit testing (the testing of single methods or subsystem using many small tests usually implemented within the main body of code) with tools such as nunit (the .NET variant of the XUnit/JUnit testing kit)

A potential way to addressing this problem is with the advanced features of Microsoft .NET framework. .NET provides two technologies that can help address this problem:

  • Attributes – The ability for the developer to add custom meta data to their program assembly that can provide information on their intentions and the use of the assembly for both static analysis and at run time.
  • Reflection – The ability of a .NET program to read the meta data of an assembly and use it to create objects and execute processes as required.
    It is possible to provide meta data on Windows controls such that a test harness can load the Windows application and ‘watch for’ an appropriate response when it triggers given events e.g. when button X is pressed then the text Y should appear in textbox Z.

Test attributes
The system can be broken into two parts. Firstly a set of attributes have been developed that allow a .NET application to ‘marked up’ for testing as shown below.

///


/// Test application with GUI testing attributes

///

[GuiTestable]
[
UserDefinedTest(“User Test 1″,”UserTest1”)]

public class Form1 : System.Windows.Forms.Form         {

       [ClickTextTest(“Button 1 Click Test 1″,”_label1″,”Hello World”)] // false

       [ClickTextTest(“Button 1 Click Test 2″,”_textBox1″,”Hello World”)] // true

       [ClickDataGridCountTest(“Button 1 Click DG Test 2″,”_dataGrid”,5)] // true

       private System.Windows.Forms.Button _button1;

 

       // implementation of other methods……

}

These attributes define where the class should be inspected for tests [GuiTestable] and the tests the developer has specified for a given GUI feature in the application. In the sample listing there are three tests defined two of type [ClickTextTest] and one of type [ClickDataGridCountTest]. In essence all these attributes tell the test harness that there is a test of a given name and when the GUI control, _button1, is pressed in the test application the test harness is to look for the stated text, or number of rows in a defined GUI control.

The test harness
A Window form Test Harness application has been developed. The Windows form application follows the same broad interface as the Nunit testing tool. The user loads the .NET assembly (.EXE or .DLL) they wish to test, using an option on the file menu. All the tests within any classes in the assembly marked as [GuiTestable] are shown. Pressing the ‘Run Test’ button will run the entire set of tests in the assembly. An indication is given for the pass or failure of each test run.. Detailed output is shown in a logger panel.

What is going on under the hood?
The key to the operation of this system is the use of reflection by the test harness. The general principle is that the harness loads a .NET assembly file using a byte stream based method to avoid locking the file to other users. It then inspects all the classes in the assembly looking for the [GuiTestable] attribute. If this is found it then further inspects all the private, protected and public members of the class for any of the testing framework attributes. Any attributes found are added to the available list of tests.
When the ‘Run Test’ button is pressed an instance of the of class object under test, a Windows form, is created as a hidden background object. Then for each test, the test harness checks the state of the Windows control to monitor for the initial value, triggers the required event on the Windows control under test and finally checks the control being monitored to make sure the value has changed as required.

Does it succeed as a testing system?
The measure of any testing system is whether the effort in setting up the tests and maintaining them outweighs the cost in doing manual tests. Many testing products fail this test; this is especially true for GUI testing. The key problem with GUI testing is that most tools rely on knowing where the controls under test are on the page. The testing model in the project does remove this requirement. The management of tests is based on the logical structure of the Windows form, not some XY grid locations. Given this fact then this testing model is a success, once the developer has added a test, hopefully, if using a ‘test driven development model’ before they implement the actual application code, then the test should remain valid for the lifetime of the control it tests. This should be irrespective of whether the control is moved, resized or any of its display properties changed.

However, this does not mean that this project offers an outright solution to all GUI testing. In essence it is a unit testing model, and all unit testing models have the issue that they test blocks of code statically. Usually a special instance of the class under test is created and the tests run. This means that often unit testing tends to be limited to testing libraries of functions that are easy to handle statically e.g. finance operations, if I have these parameters do I get the correct numeric answer? This becomes a problem for this GUI testing models, no GUI is static; its job is to dynamically display information generated by backend business logic. Care has to be taken when designing tests that the Windows form when created does have the required underlying business features, such as connections to databases, to allow meaningful tests.

Can’t wait to get your hands on it ?

The next step is for me to package you my test system such that it is easily usable and extensible by others. This should be done in the near future, other projects permitting! Keep an eye open on this blog for further postings

 

 

 

?>

Another seminar, this time on Web Applications

Today we ran our second spring seminar, this time on technologies to enable Web Business. As usual the white paper associated with the event can be found at http://www.blackmarble.co.uk/WhitePapers/2005/Seminar Notes – Web Applications for Business.pdf

In the morning we discussed how ASP.NET, and specifically the new features in VS2005, can provide a very productive environment to develop both B2C and B2B systems. A special focus was made on how these tools can be used to create truly accessible web sites.

The afternoon focused on webservices, and was really an introduction to our final spring seminar on ways to best use data within an organisation. At this event we will be looking at Delivering Data using Web interfaces and managing data with Reporting Services and BizTalk. The event is free, go to http://www.blackmarble.co.uk/Events/May2005.aspx to find out more.

Compiler Dev Lab last day

It is the last day and so far it has been a great ammount of fun. This morning the SQL team
came to discuss the implementation of the CLR in SQL Server 2005.
The most enjoyable part of the day was a round table discussion.

Robert

DevLab Tuesday Evening

Jeff Sanquist organised a BBQ which was a trailer in the car park (remember it is February Folks) manned by a Hairy Biker; it must be said there was a point where I thought Jeff had lost the plot, I should have had more faith, the food was great. When I ambled down to the lecture room Channel 9 had just started filming a chat. UPDATE the video is now on Channel9 I am lurking in the background of the video and show up on the still :):)

Robert

Dev Lab Tuesday

Today we started with a presentation on VSIP program,
Chris Lovett presented “Whidbey XML editor and language service framework”,
the XML editor and XSL debugger (isn’t Whidbey just great) are shipped as standard
and Chris showed how this was done.

One of the most interesting parts showed how Visual Studio provides both
colourization and intellisense services. One of the most surprising is that
the colourisation is only passed characters on the current line and it is
expected to remember the rest of the line.
As visual studio requires parsing every time the system changes, parsers
need to be fast (in reality can only be lexers).

Kieran Mockford who we all remember as giving some great demonstrations in the UK
over the last few years is now a program manager for MSBuild.
As you would expect Kieran’s demonstrations were spot on. There was a lot of interesting facts
presented about whidbey and MSBuild but …

A great presentation by Shon Katzenburger who is the technical lead on the C# compiler
team, on Implementing generics in your compiler, really made me think about Generic
inference rules. If you are planning to use generics make sure you understand the rules
of inference so you can guarantee which methods are available in the applicable method
pool.

Side Note You need to be careful when defining generics Struct G<T> { static G<G<T>> g;> will overflow the stack,
I guess I hadn’t thought of even doing this due to the standard rule of don’t be too clever when you are doing
a design, but it is useful to know.

Jim Hugunin presented on Dynamic Languages on the CLR, Jim started to produce Python on .NET to prove that .NET
did not work as well as other down grade runtimes :), however after proving to himself and others he was wrong he
now works in the CLR team helping others see the power of .NET.

Microsoft Compiler Dev Lab

I am over at Microsoft in Redmond attending a Compiler Dev lab.
Compiler developers for all sorts of interesting compilers are here producing Cobol, RPG, Eifel, Delphi and others.

The event is run under NDA, so there is unfortunately a large number of ommisions.

John Gough provided an overview of producing your first CLR compiler,
I have seen John Gough present before and I have read the book (no tee shirt just yet).
The presentation was very well done and he managed to cover a lot of material,
If you are interested in compilers in .NET read his book.

Jim Millar presented the “Rules of the CLS” which was a fully interactive
discussion on the published rules of the CLS. Members of several teams at
Microsoft CLR, Visual Basic, C# and C++ all contributed to the discussion 🙂

Robert