Ship It! - 2.7 Use a Test Harness

A testing harness is the tool or software toolkit you use to create and run your tests. 

Tip 9: Exercise your product - automate your tests

Try to use a common testing framework.  Make sure it can be driven from the command-line. Take a peek at MetaCheck to see if would help your team.

Tip 10: Use a common, flexible test harness

Unit Tests are designed to test your individual class or object. They are stand-alone, and generally require no other classes or objects to run. Their sole purpose in life is to validate the proper operation of the logic within a single unit of code.

Functional Tests are written to test your entire product’s proper operation (or function). These tests can address your entire product or a major subsystem within a product. They test many objects within the system.

Performance Tests measure how fast your product (or a critical subsystem) can run. Without these tests, you can’t tell whether a code change has improved or degraded your product’s response time (unless you are really good with a stopwatch!).

Load Tests simulate how your product would perform with a large load on it, either from a large number of clients or from a set of power users (or both!). Again, without this type of test, you can’t objectively tell if your code base has been improved or degraded.

Smoke Tests are lightweight tests and must be carefully written to exercise key portions of your product. You would use smoke tests because they run fast but still exercise a relevant portion of your product. The basic idea is to run your product to see if it “smokes,” i.e., fails when you invoke basic functions.

Integration Tests look at how the various pieces of your product lines work together. They can span many products: sometimes your products and sometimes the third-party products you use.

Mock Client Testing is used to create tests from your client’s point of view. A mock client test tries to reproduce common usage scenarios for your product, ensuring that the product meets minimum functional specifications. This type of testing can be very effective for getting essential testing coverage in place to cover the most commonly used code paths.

How Do I Get Started?

• select a testing tool or toolkit
• start adding tests to problem areas
• ensure your tests are being run as part of the build system

Am I Doing This Right?

• are your tests effective? Are you catching bugs?
• What are your code coverage numbers?  Are they increasing over time?
• is your product testable?
• do your tests tell you whether they pass or fail?
• does everyone in the shop have the ability to add tests?

Testing is important and it seems like everyone has their own definition for the different type of tests.  I'm comfortable with the definitioins the book presents.  I've seen discussions around testing your system via Archetypes.  Test your system like you are a power user who uses more features than the normal user.  Test your system like you are a first time visitor just browsing the landscape.  I think Mock Client Testing is along those lines of thinking.