CS 305: Usability Testing


Highlights of this lab:

  • What is Usability Testing
  • Designing Pre-Task, Task, and Post-Task Questionnaires
  • Performing Usability Tests and Measuring User Performance
  • Assignment

:: What is Usability Testing ::

Usability testing is one of the most fundamental methods in usability engineering. Usability testing is the process of testing your design with system users. Usability testing gives system designers a means of finding out whether their system meets its intended purpose.

How Many Users Required?

There is strong evidence that you need only up to 5 users in usability testing. There are some exceptions although - it all depends on what you are trying to find/accomplish in your examination. Neilson has offered the following graph to illustrate how and why:

When we observe the above curve we notice that it is obvious that zero users gives zero insights.

As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.

When you test the second user, you may discover that this person does some of the same things as the first. However, you may learn some new things...and so on for the third, fourth, and fifth person. After the fifth person you already know approximatemately 85% of the system errors. As you add more and more users, you learn less and less

Although the curve clearly shows that you need to test with at least 15 users to discover all the usability problems in the design, because of the cost a and time needed you may use a smaller set. Also, if you need more users, you can have 3 separate tests of 5 users each and so on.

Importance of Pilot Evaluations

Pilot evaluations, such as the one we are going to be doing in this lab, are very useful in understanding what aspects of your usability evaluation works. For example, after the pilot evaluation you may find that you need to add a few more questions to one of your questionnaires, etc.

Elements of the Test

Depending on who you talk to, there are different elements for every usability test. The main parts of the test are however:


:: Designing Pre-Task, Task, and Post-Task Questionnaires ::

Let us look at the files for each of these kinds of user tasks:


:: Performing Usability Tests and Measuring User Performance ::

Here is a rough guidline to how you should conduct a usability test:

Measuring User Performance

There are many ways you can rate and grade user performance. While, and after, your users are completing the task questionnarie you can:


:: Assignment ::

The first part of this lab is an in-lab exercise designed to give you hands-on experience conducting a usability evaluation testing. You will be placed into groups of 2 and each of you will take turns conducting a pilot evaluation.

For the second part of the lab, after you are finished conducting the pilot evaluation, I would like you to write a summary of your pilot evaluation experience, discussing what worked well, what didn't work well from both the evaluator and the evaluatee (not sure this is a word....but it is now! ;-). Comments could include additions to the study documents, a formal method of rating the users performance while using the systems, etc. I would also like you to include all of the scores (performance-scores and time-scores, comments) acquired by each evaluatee.

This lab will be worth a total of 2%.

Hand in your summary using Web-CT. This is due by the start of your next lab


:: Online Source(s) ::