Java Testing and Design - 1.5 Testing Methods

This section describes the following testing methods:

• click-stream testing
• unit testing (state, boundary, error, privilege)
• functional system testing (transactions, end-to-end tests)
• scalability and performance testing (load tests)
• quality of service testing

Click-stream testing is when you keep track of the clicks a user performs as she navigates your sites.  The thinking is that more clicks equals more advertising revenue.  This type of testing doesn't tell you if the user was able to achieve his goal when visiting your site and isn't very useful when trying to determine the quality of your software.

Unit testing is when a developer tests an individual software module by providing it various inputs and comparing its against expected outputs.  Although useful to developers, module based testing doesn't tell you how useful or healthy the entire web application is.  Don't rely solely on unit tests when assessing your system's health.

Functional system testing takes the perspective closer to that of the end-user in that they check the entire application from end-to-end.  A system test will start at the browser, travel through the web servers, individual software modules, the database and then back.  When system testing, you have to understand your system well enough to know exactly what front-end actions trigger interactions between the various back-end components.  Bringing up a page which is a simple static HTML isn't much of a test because it doesn't trigger software modules to fire that access the database and other integration points.

Scalability and Performance testing tries to understand how the system behaves with concurrent users accessing the system.   Typically, the functional tests can be used to drive the system and obtain loaded system performance metrics.

Quality of Service testing attempts to show that the application is meeting service level agreements.  Functional tests should be used to monitor the web application's health.  Done over long periods of time, you can get a picture of what is normal and what is not.

A test agent is a framework or process that allows for automation of the system's test.  Typically, an agent must provide the following:

• checklist - defines the conditions and states of the test, such as being able to log into the system
• test process - lists the system transactions that are needed to perform the checklist
• reporting method - records the results after the process and checklist are completed

The more automated a testing agent is the easier it is to certify the health of the application.

You can use your automated test agent to test for scalability and performance.  Scalability describes the system's ability to serve users under varying levels of load.  Run your tests when the system has 1, 100, 500 and 1000 concurrent users and measure how long the tests take to respond and you have an idea of how scalable your system is.  Performance measures the system's ability to deliver accurate results under load.  If the system responds quickly (scales) but returns incorrect results or errors it isn't performing very well.  Taken together, performance and scalability can tell you how your system behaves under load.

Creating functional tests can be difficult because it can be difficult to know exactly how the tests should interact with the system.  One idea to try are Archetypes.  Construct several testing "personalities" which describe different types of system users and capture the way they interact with the system.  For example, a field agent might be somebody who does lots of reads and relies heavily on the querying capabilities of the system.  A manager might be more of a power user who uses the reporting portion of the system to generate his monthly status reports to his boss.  The more detailed archetype you can create the better your test agents will be and the better tested your system will be.  In practice, automation of the test agents is the only practical way of running the tests in any consistent and meaningful way.