Pattern 7: Usability testing **
AKA:

You have built a version of the site and completed some AUTOMATED TESTING (6) and perhaps tested against TWO-YEAR OLD BROWSERs (10). You have ESTABLISHed THE USE CASES (3) and you understand the object model. However...

You just cannot test things like the consistency of left/right mouse button usage, USE OF COLOUR (53) or many other aspects of usability automatically.

Therefore

Consider hiring a usability consultant from outside the organization to avoid personality or political conflicts.
Perform usability tests from the first prototype continuously throughout the project. Do not confuse usability tests with output from focus groups. If formal, lab-based tests are not within the budget then test informally with the developers as silent observers. If you can't get real users grab people 'off the street' or from other departments or offices. Video the session if possible. Ask people to think out load as they use the site. Record their comments. First ask them to browse and react unprompted then give them tasks based on the use cases. Record their successes and failures. Ask them if they're happy as a result of visiting the site - and why. Reward them for their trouble and ask if they would help in future. Don't forget to test on dial-up lines or under conditions that simulate them accurately. Use automated tools such as Prophet to enhance your understanding of how users behave.

Remember KISS (38) - resist the urge to add features, such as help messages and explanations, in response to testers' comments. Instead try to remove features that might have confused or distracted the users. If possible, simplify the navigation using the appropriate patterns downstream from this one.
If the testers were able to get back on track easily after making an error then the cost of fixing it may not be worthwhile. Fix it only if it is easy to do so.

Change manage requests for new features as you would in any software development project; i.e. make sure there is a justifiable benefit.

Video the test sessions to record both what was on the screen and the user's actions. Record users' comments and try not to lead. Ask open questions (questions that can't be answered with a yes or a no). Record comments. Let the developers watch - possibly on a screen in a nearby room - but don't let then interfere, criticise, help or explain.

Although there is much more to learn about the topic of usability testing, this pattern is terminal within this language. However, make sure you also do GET-IT? (8) testing in parallel with this pattern.

Reviewers and sources
Frank Buschmann, Kevlin Henney, Nora Koch, Oliver Vogel, Uwe Zdun

The first part of usability testing is based on the use cases. Users must be able to perform all key tasks successfully and without frustration, long delays or using tortuous navigation around the site. The use cases help to define scripts for this kind of test. The tests have a dual aspect: did the user accomplish the task and did they find it easy and pleasurable to do so?
Focus groups may be useful before you begin design. They will help to establish objectives and use cases but they are no substitute for usability testing. You can start the latter as soon as you have even an outline design in the form of hand-drawn screen mock-ups or storyboards. Usability testing should then continue throughout development. If you do it at the end of the project it will be too late to fix the defects that it uncovers.

Obviously, it is better to test with real users rather than or as well as surrogate ones. Unfortunately, for web sites it is very rare even to be able to know who they are, never mind test with them. When all else fails technology can come to the rescue. Prophet (www.speed-trap.com) is a product that enables site designers to observe the actual behaviour of users as the visit the site. Prophet works by downloading an applet to the browser of every visitor. This applet then monitors the users' mouse movements and clicks and keyboard operations. This delivers a vast amount of information about usability - albeit indirectly. Consider, for example, a user filling in a form. If they spend an average of 1 second typing into each field but pause over the fifth one for 40 seconds, one could infer that they have had to leave their desk to retrieve some piece of information asked for: a possible usability problem. Or do they get as far as the home page and just go away: a definite usability problem. If they move around the site a lot you might be able to infer that they are confused by the navigation. Prophet is therefore a great way of magnifying knowledge about a site's usability - although it is not a substitute for tests.

Speed-trap insists that its customers display a privacy notice permit the user to refuse the download. However, the information it gathers is entirely anonymous and the applet, being pure Java, cannot see what the user is doing outside the browser window. Therefore one would hope that most users would find this monitoring acceptable. Speed-trap estimates that their system misses about 20% of users either because they demur or because they have Java disabled or use an old browser version.
Rainassure from Rainfinity is another product that can monitor transactions on e-commerce sites but runs on the server and supplies less usability data although it does spot transaction failures that could provide some clues. One news site (telegraph.co.uk) uses a company called Whitecross to track user behaviour offline. The information is used mainly for audience analysis but, again, could provide some clues as to usability.

This pattern conceals much complexity and there is a great deal more to say about usability testing of web sites. There could well be an entire pattern language for this topic. Useful sources to consult further include Lindgaard (1994), Cato (2001), Krug (2000) and van Duyne et al. (2002).

Browse the language	What is Wu?	Look at an example pattern sequence	Structure of the patterns
Comment on Wu	Contributors	Return to TriReme home page	Links to related sites