UX Design 7 min read

An Introduction to Quantitative and Qualitative Usability Testing in UX Research

Quantitative vs qualitative usability testing

The success of a product depends on the satisfaction level of the end-user/customers. This is true for every product and every market. Though the producers strive to provide the ultimate satisfaction or user experience to the customers, how do they foresee how successful a product is going to be? 

The only way to predict usability is to conduct usability testing of the product and modify the product based on the test reports before releasing it to the target market.

By definition, usability testing is evaluating or assessing a product or service by actual users. Often it is also used or conducted to gather data on users. But what happens during a typical usability test? 

Examples of quantitative vs qualitative research methods

Source: Matt P Lavoie

During a typical usability testing, the evaluators will try to complete tasks or accomplish a target with the test product and they will be observed and recorded by observers or people who are conducting the test. The data are collected through both quantitative and qualitative processes and analyzed.

Thus, usability testing is used not only to gather data on the product and its usability but also on the users’ ability to perform or their efficiency levels with tasks. 

Quantitative vs. Qualitative Usability Testing

Quantitative and qualitative user testing process

A usability-testing study involving users or participants who are assigned to perform one or more tasks can also be called a user-testing study. In such testing, two types of data are collected – qualitative data and quantitative data. Qualitative and quantitative data are not generally collected at the same time and require different study setup and analysis methods. But both are essential in the iterative design cycle. They should not be used interchangeably and it is important to know when to use which type of study and why.

Though both forms of data are collected, user testing methods are more qualitative than quantitative.

Questions answered by different research methods in quantitative and qualitative  user testing
Questions answered by different research methods in quantitative and qualitative user testing

#1 Nature of data

Qualitative data  

This consists of observational data on user perceptions, opinions, and feelings. This kind of data offers a direct assessment of the system or the user. Observers can identify not only how hard or easy the tasks are, but also the exact part of the task or design feature that is easy or hard. The observations can be augmented through the follow-up questionnaire.

Quantitative data 

This data can be calculated or measured like time taken to complete parts or tasks. This data offers an indirect assessment of the usability of a design or the users’ performance. Task-completion time, the number of errors or success rates can only provide a perception of usability. A common example is satisfaction ratings for a product. 

As quantitative data are only numbers, they are not used to interpret a usability testing study in absolute terms, as they don’t tell about the problems that arose or suggest a modification to the design for better performance. This is also true from a user or performance point of view.

#2 When to conduct a Qualitative study

  1. A qualitative study provides answers to ‘why, ‘how’ and ‘what’ like – ‘why did you stop…?’ ‘What happened…?”How did you feel…?’ As a qualitative study brings the user and the test team closer a more personalized study can be conducted. There can be an actual conversation and the team can gain useful insight from the users. These insights can be used to modify /better the design.
  2. To assess the performance of a participant in a user-testing study. The insights from the participant or the data can be put into context and the performance can be evaluated more accurately.
  3. A situation where the participants or users are limited in number. A qualitative study can provide in-depth insights, it requires fewer participants or users. 

#3 When to conduct a Quantitative study 

  1. This study throws light on group behavior but doesn’t talk of the individual participants. It can identify the percentage of participants having a problem with a particular task but won’t tell the individual problems. This objective nature of data helps in identifying areas where things go wrong without talking about what went wrong.
  2. They are used to validate the usability of a larger project. For example, a payment app will require validation in user ratings and percentages.
  3. They require a greater number of users or participants as the data obtained is in the form of numbers or measurements. A larger user or participant group can provide a substantial figure to validate a study.

#4 Qualitative testing methods

Moderated vs. Unmoderated (mediated vs. unmediated) testing 

A moderated or mediated test means a study based on direct interaction between the participant/user and the moderator. The moderator hears what the participant feels and thinks and probes more to get the detail out. There is a chance to observe and record participants’ body language and expressions while performing the task. The level of intervention can be at the moderator’s discretion. This kind of test is time-consuming and also requires skilled manpower.

Unmoderated or unmediated tests don’t require direct interaction and can be conducted on a larger group of users or participants. The users /participants are given certain tasks and are observed and recorded while they are completing the task with minimum interaction.

Tree Testing and Card Sorting

Tree testing: user testing method
The test will show the user a small sample architecture of the product and give them a component in that architecture. Then they are asked to place them correctly in that architecture.

Tree Testing is used to assess the general logic of the structure of the product/service. The participants are presented with a simplified visual representation of the product architecture (which is usually in the form of a tree). They are then given an item representing a part or a piece of information and told to place it in the appropriate position (branches) of the logic tree. The time spent or the doubt arising in labeling and sorting of the information pieces are recorded and later analyzed. If the participants show greater confusion in categorizing a particular piece then that is red-flagged.

Card sorting in usability testing
Users are given a set of cards that are parts of product architecture. They are then asked to group these cards according to their needs.

Card sorting is the opposite of tree testing. Here the participants are given a set of information and asked to create a logical structure out of them.

This kind of test not only points to what kind of structure makes more sense but also to the user’s mental set up and reasoning abilities.

#5 Quantitative testing methods

A/B vs. Multivariate Testing 

In A/B testing, the participants are presented with two versions of the same problem/ task and asked to complete them. It is a simple way to find out the right direction but works when there are very limited solutions.

Multivariate testing is essentially the same as A/B testing, but here instead of presenting two choices, several are presented at the same time. This is a practical way to narrow down choices.

Eye Tracking and Heatmaps 

Heat mapping and eye tracking in user testing
Heat mapping and eye tracking usability tests

Eye Tracking is a powerful quantitative technique. This test identifies what a participant/user focuses on most. For example, when interacting with an interface, the participant is attracted to a particular component and stays on it for some time to process it completely.  This movement can be tracked and measured to provide a trend or an understanding of the bigger picture. This works better when used and compared with similar prototype designs. Unlike Eye Tracking, Heatmaps point to those parts of the interface or design where the participants focus on the most. The quantitative data thus tell which parts of the design the participants found most interesting. Translation of the data makes a heat map pointing to all the hot areas in the design.

#6 Statistics in qualitative and quantitative studies

One significant advantage of quantitative studies over qualitative ones is the use of statistics. Since quantitative data is mostly numbers that can be measured, they can be presented in a more orderly way with the help of statistical tools. 

When quantitative research is presented in this manner, it doesn’t seem random or biased. They help to see the trend/deviation among the users or participants from an objective point of view. They also help to summarize the findings in a more orderly manner and help to predict or point towards a hypothesis or negate it.

As qualitative data can be observed or heard, it is descriptive and can’t be represented through statistical tools. The qualitative data also cannot be assured as truly objective and represents the whole.

Conclusion

While testing a product/design or service, both qualitative and quantitative studies are important and used. Both of them can reveal the weak spots in the product and help to overcome them. 

Qualitative studies are more direct as they get into the finer details of functioning and user experience. It is an effective tool to understand the main usability problems in a design and can be efficiently used to bring the necessary changes required.

On the other hand, quantitative studies, though indirect, offer a summative evaluation and point towards a definite direction. It is important to understand the strength and weaknesses of both the methods and more importantly to understand when to use a particular method. 

It is important to understand the objectives of both types of testing that demonstrates the need for continual and iterative testing. User experience research is a continuous process of learning and improvement and should be conducted in a cycle to derive optimum benefits.

Topics in This Article:

Usability testing user centered design UX Design UX research
Ray Slater Berry

WRITTEN BY

Ray Slater Berry

Ray Slater Berry is the content lead at Skale. He has been working in social media, content marketing, and SEO for nine years. He specializes in the tech, innovation, design, and product sectors. He is also a published psychological thriller author with his first novel, Golden Boy.

You might also like