HCI Forum topic DISS 720

Below are Carol Barnum's suggestions on how many users it takes for a (web) usability test.

5 starter questions:

What is your testing goal? (You need to know this first to answer the other questions)
How many users do you need to reach your goal?
How do you define your users?
What’s the basis for small numbers? How small is OK?
The numbers controversy: Do small numbers work for web usability testing?

--------------------------------------------------------------------------------

Testing goals:

To find problems with the product
To eliminate problems with the product
To improve user satisfaction
To determine if you made the right assumption about user tasks
To help developers understand the user experience
To learn from prototypes
To determine the level of proficiency (e.g. time on task)
To understand the Learnability of the product
To determine quantitative issues
Numbers needed to reach goals:

For summative (product completion testing, where statistical validity is needed) Need 25 to 30 or more users
For exploratory testing, can use Nielsen model of 4 to 5 users (even as small as 3 users)
Numbers depend on the kinds of users (users must match in experience, knowledge)
Must select specific subset of user population; must use scenario-based tasks; for diagnostic purposes; part of iterative process
Controversy--Why web testing may be different:

Jared Spool tested ecommerce sites
Users given money with goal to find something to buy (open ended task)
Rolf Molich directed comparative usability evaluation of Hotmail
9 different labs set up different goals and users
Both studies showed that it takes many more users than 5
Question: can you test websites with small number of users?
Answer: Yes, when you
Use scenarios
Screen users to match specific subset of user population
Question: how do you know what’s generalizable?
Answer: When the following conditions are in place:
Random sample with small numbers
Common skill sets of users (e.g. level of domain knowledge)
Same motivation from users (desire to reach task goals)
Data may not be generalizable—not applicable to any user
However, findings for different tasks can point up the same problem
Issue may be information architecture
What types of users to recruit?

It depends on business goals.
If numbers are very small, try not to mix the users.
Better to go for one end or the other on the continuum; avoid the middle.
"Most people" is not a good description of the user.
Should you separate the documentation from the product?
Test documentation by itself
Test product without document to see where users need it
What to do with one user’s findings (outlier data)
Examine the issue for severity
Study the screener for particulars about this user
Ask yourself: can you envision other users having this problem?
Have you seen this in other studies?
What is the importance of this user (validity—a big customer)?
How to validate the findings with small numbers

Talk to tech support
Talk to training
Talk to sales/marketing support (but be cautious, because they may not be talking to users)
Know what the product is supposed to do
Know what the product can deliver
Reporting results

Capture the "eureka" moment
Shun percentages when sample size is small (say, instead, "several" users or "three out of five" users)
Create a matrix of tasks
Indicate success/failure
Time
Questions asked
Sample comments
Errors and severity (recovery from errors should also be noted)
Distinguish confidence levels in findings
Descriptive information is also useful (e.g. user confusion)