GUI Testing Framework - First software release

author Black Marble calendar Aug 9, 2005 · 13 min read · Uncategorised  ·
Share on: linkedin copy

1 INTRODUCTION

1.1 The Problem

The graphical user interface (GUI) of a computer system is one of the most troublesome sub-systems to test. Though some products have been marketed to address this area they have not been that well-received or effective. All such products have tended to be based on macro recording systems that often introduce more errors than originally present in the software they aim to test. The net effect of this is that the test engineer spends more time debugging their test scripts than they do testing their product, as the slightest change in a GUI layout will usually break the test harness. Hence up to now the most effective way to test a GUI is to get human testers to do it, though this is not without problems of its own.

However these problems can be avoided when testing non-GUI components; a variety of excellent tools and techniques exist to address this area such as JUnit and NUnit as well as commercial products from companies. As in many applications the GUI can be separated from the business logic, these tools can be used to test many features of a system. However, the problem remains that these products are not directly applicable to the testing of GUIs.

1.2 A Potential Solution

The lack of GUI testing is a major, but currently unavoidable, hole in current software engineering processes. Only so much can be done when separating the GUI from unit testable backend functions.

This sample shows how this problem can be tackled with the Microsoft .NET framework. .NET provides two technologies that help address this problem:

  • Attributes – The ability for the developer to add custom metadata to their program assembly that can provide information on their intentions and the use of the assembly for both static analysis and at run time operations.
  • Reflection – The ability of a .NET program to read the metadata of an assembly and use it to create objects and execute processes as required.

The basic method is to provide metadata on Windows controls and events such that a test harness can execute the Windows application and ‘watch for’ an appropriate response e.g. when button X is pressed then window Y should appear.

2 TOOLS USED

2.1 Microsoft Visual Studio 2003 and C#

All code is written in the Visual Studio 2003 version of C#. This requires the .NET framework 1.1.

2.2 FXCop

FXCop  is a free tool for Visual Studio 2003 provided by Microsoft, and built into Visual Studio 2005. It provides a means to automatically check that .NET code is written to current recommended design and style guidelines. It can be extended with a developers own rules if required.

In this project the default rule set has been used.

2.3 Automatic XML Documentation

All public fields, properties and methods within the project are documented using the C# XML documentation system.

This XML documentation is used to build a documentation web site using built in tools within Visual Studio and a standard Windows help file using nDOC .

2.4 Source Control

All software source code was stored under Microsoft Visual Source Safe revision control software during development.

The code provided in the sample package should have had the links to VSS removed.

3 USAGE OF THE TEST HARNESS

The basic tools for the system are provided in the GUITesterSampleV1.zip file. The full source code is provided in GUITesterSource-V1.zip, and the user manual in GUITester Usage Manual.pdf

3.1 Marking Up Application for Testing

Attributes are applied to an application in a similar manner to nUnit.

A fully marked up sample application has been provided.

3.1.1 Marking a Class as Testable

To mark a class as testable as single attribute needs to be applied to the class definition.
 [GuiTestable]
 public class Form1 : System.Windows.Forms.Form {…}

Once this is added the class should appear in tester application when the assembly is loaded (see section 3.2).

3.1.2 Defining Tests

The following test types are available

3.1.2.1 ClickDataGridCountTest

This test type can be applied to any GUI control that issues a click event. It counts the number of rows in a DataGrid control.

It is applied to the definition of the GUI control as shown below:

  [ClickDataGridCountTest("Click DG Test 1","dg",5)]
  private System.Windows.Forms.Button myButton;

or

  [ClickDataGridCountTest("Click DG Test 2","dg",0,5)]
  private System.Windows.Forms.Button myButton;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The name of the target DataGrid
  • The initial value to expect before the click event (optional)
  • The final value to expect after the click event

3.1.2.2 ClickOpenFormTest

This test type can be applied to any GUI control that issues a click event. It checks to see if a suitably named window is opened then the click event occurs.

It is applied to the definition of the GUI control as shown below:

  [ClickOpenFormTest("Open Window Test","Form2",true)]
  private System.Windows.Forms.Button myButton;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The Window name of the forms that should be searched for.
  • Set to true of the Window should be closed once it has been detected

3.1.2.3 ClickOpenMsgBoxTest

This test type can be applied to any GUI control that issues a click event. It checks to see if a standard MessageBox has been displayed with the expected message.

  [ClickOpenMsgBoxTestAttribute("Click Msg Test","Message")]
 private System.Windows.Forms.MenuItem myMenu;

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The message text expected

3.1.2.4 ClickTextTest

This test type can be applied to any GUI control that issues a click event. It checks the contents of any GUI control that supports the .Text property.

It is applied to the definition of the GUI control as shown below:

[ClickTextTest("Click 1","textBox","Hello World")]
  private System.Windows.Forms.MenuItem myMenuItem;

or

  [ClickTextTest("Click 2","label","Before",”After”)]
private System.Windows.Forms.Button myButton;

The parameters are:

The name of the test (for the benefit of the developer, not used by the testers in any way)

  • The name of the target GUI Control that contains the text
  • The initial value to expect before the click event (Optional)
  • The final value to expect after the click event

3.1.2.5 TextSizeTest

This test type can be applied to any GUI control that contains text in a .Text field. It checks that the text currently displayed is fully visible.

It is applied to the definition of the GUI control as shown below:

  [TextSizeTest("Size test")]
 private System.Windows.Forms.TextBox textBox;

The parameter is:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)

3.1.2.6 UserDefinedTest

Unlike all other tests, this type of test is applied to a class, not a GUI control.

[UserDefinedTest("User Test","UserTest1")]
 public class Form1 : System.Windows.Forms.Form {…}

The parameters are:

  • The name of the test (for the benefit of the developer, not used by the testers in any way)
  • The name of the method to run as the test
  • The test method should be in the form as shown below, returning true if the test is passed.

This method can contain any valid .NET code.
///

  /// User defined test
  ///
  /// True if passed
  public bool UserTest1()
  {
   return false;
 }

3.2 Graphical Testing Tool

Load the GUITestHarness.exe (found in the GUITesterTestHarnessbinDebug directory). This tools follows the design model of the tester in nUnit.

Use the load option on the file menu to open the sample application for testing. You should see the available tests in the tree control.
 
The tests can be run by pressing the ‘Run Test’ button or by using the Tests menu. In both cases an option is provided to run one or all of the tests.

3.3 Command Line Testing  Tool

The command line tool is run in the following way from a DOS box

_ConsoleTestHarness.exe MyAssembly.DLL
_ 
The only differences from the GUI tester are:

  • That the command line creates a new instance of the object under test for each test in turn. This may need to be reviewed in the future.
  • All tests are run; there is no way to control the running of a subset.

The command line tool provides a simple output to the StdOut (the command line) but also returns a DOS error level for success for failure.

4 REVIEW OF THE PROJECT

4.1 Is the result worth the effort?

The measure of any testing system is whether the effort in setting up the tests and maintaining them outweighs the cost in doing manual tests. Most testing products fail this test; this is especially true for GUI testing. The key problem with GUI testing is that most tools rely on knowing where the controls under test are on the page. The testing model in this project removes this requirement. The management of tests is based on the logical structure of the Windows form, not some XY grid locations.

Given this fact this testing model is a success. Once the developer has added a test, hopefully, if using a ‘test driven development model’ before they implement the actual application code, then the test should remain valid for the lifetime of the control it tests. This should be true irrespective of whether the control is moved, resized or any of its display properties changed.

4.2 Is the system dynamic enough?

However, this does not mean that this project offers an outright solution to all GUI testing. In essence it is a unit testing model, and all unit testing models have the issue that they test blocks of code statically. Usually a special instance of the class under test is created and the tests run. This means that often unit testing tends to be limited to testing libraries of functions that are easy to handle statically e.g. finance operations, if I have these parameters do I get the correct numeric answer?

This becomes a problem for GUI testing models, no GUI is static; its job is to dynamically display information generated by backend business logic. Care has to be taken when designing tests that the Windows form does have the required underlying business features, such as connections to databases, to allow meaningful tests.

4.3 Is it usable for a developer?

Given all these facts this is a viable testing system, but whether it would be taken up by the .NET developer community is another question.

The NUnit project, since the inception of this project, has released a system called NUnitForm . This tries to address similar issues to those addressed in this project, but building on the NMock  system to create stub instances of any business and GUI classes required.

Due to this architecture, NUnitForm becomes a very comprehensive and flexible system, but fairly complex to use. The system defined in this project is far simpler to use, but may be too inflexible for practical use because of this simplicity. However this project’s relative simplicity may be its greatest advantage.

Also any product based on NUnit has a good degree of momentum within the development community.

4.4 What are the current limitations?

In its current form this framework has some significant limitations. However these are not fundamental, they are just issues that have yet to be fully resolved. With a suitable application of time and effort they can be addressed.

4.4.1 Object Hierarchies

The test harness should be able to explorer the hierarchy of classes within an assembly. This requires that the test discovery process should use a tree walker model as opposed to a simple list model.

4.4.2 MDI Forms

Not all Windows applications are SDI, many use a MDI format. The test harness should be able to test MDI applications. This is related to the issues raised in 4.4.1.

In MDI applications there are two types of form: an MDI parent (the container) and MDI children (the forms within the container), both should be testable.

In most cases testing of the individual MDI child forms, running as temporary SDI forms will suffice. However this does not allow for testing of inter-form MDI form interactions. To do this further work will need to be undertaken using the Windows API to explore the inner contents of an MDI application using its various Window handles

A mechanism for testing the interrelation of child MDI forms would greatly add to the possible uses for the system.

4.4.3 Application object

It was noted that use of the Application object was not possible in the test harness. One of the side effects of this problem is that the standard means for providing user configuration information is not available in the test environment. The standard .NET means to provide configuration information is to use an XML file, by default if a program is called MYAPP.EXE the configuration file is called MYAPP.EXE.CONFIG. This is looked for in the same directory as the .EXE.
A partial solution has been provided by the use of a wrapper class when trying to use resources in the directory that contains the assembly under test.

If a full solution can be found to the discovery of the path to the assembly under test, then the testing framework becomes far easier to integrate with existing applications as fewer changes in the application code will be required.

It is thought that this is the one limitation that will be the hardest to resolve.

4.4.4 Test Order & Setup

The lack of a flexible means to order tests is a major limitation at this time. The knock-on effects of one test can cause another to fail when it should not. This can be resolved by the addition of means to define setup and rip down methods, and a more formal way to order tests or, even better, a means to sub group them.

4.4.5 Threading

The tests with MessageBox’s showed that for any modal Windows operations a multi-thread model was unavoidable.  Migrating the testing tools to a full multi-thread model should be considered a priority. Currently multi-threading is only used for the MessageBox test. However a developer using this framework should not have to worry whether they application is modal or not. The test harness should be able to handle the form irrespective of modal issues.

5 POSSIBLE FURTHER ENHANCEMENTS

The current list of areas for possible further enhancements is:

  • Counting the number of items in a ListBox, probably similar to the DataGrid test.
  • Radio buttons click – need to check how having the radio button in a group (so only one radio button can be checked at any one time) effect the tests
  • Do group boxes and panels within a form effect how controls are found? It is assumed that the use of the GetValue call may have removed this as an issue
  • Initial state and test order. Do we need a set of [setup] attributes (ones to run once for a test run and ones to run before each individual test) to prepare the application for testing?
  • Do we need to force a test order?
  • It is not possible to load a running .EXE into the test harness? Currently, due to the way assemblies are loaded, they are locked by the operating system. Is there a way round this problem?
  • How best can the MDI children, inside an MDI container, be tested?
  • Can the set of tests be nested within the test harness using the sample applications class hierarchy?
  • Make the testing system multi-threaded

This work is copyright © Black Marble Ltd.