What To Know About Usability Testing Before You Start
Usability testing gives you hard data on how real users navigate your design, showing where they struggle, slow down, or drop off.
With this insight, you make decisions based on user behavior, not guesswork or assumptions.
And usability testing is not just about improving the user experience (UX), it’s about mitigating risk. A confusing site drives users away, kills conversions, and damages your brand. Testing helps you catch these issues before they cost you customers.
What is Usability Testing?
Usability testing is a structured way to evaluate how real users interact with websites, mobile apps, software, or other digital products.
The goal is to uncover issues in the design that frustrate, confuse, or trip up users, as well as potential opportunities to improve user satisfaction, efficiency, and confidence.
At the center of every type of usability testing there are three core components:
- Test: A carefully designed and controlled evaluation where participants attempt predefined tasks to assess usability.
- Facilitator: The person who guides the usability test, observes user behavior, and asks follow-up questions to uncover qualitative data.
- Participant: A real user who interacts with the product during testing, attempts tasks, and provides direct feedback (e.g. verbal comments, survey responses, or ratings) and indirect feedback (e.g. hesitation, errors, time on task, or click patterns).
There may also be a defined role for observers who watch the test (live or recorded) to gain perspective, but don’t interact with the participant.
To provide a simple example, a usability test might focus on the checkout experience, and a facilitator would observe a participant as they tried to make a purchase. If they struggle entering information, figuring out where to select, and make errors that slow down the process, the facilitator is going to have tons of ideas about specific areas of the design that need attention.
The capabilities and limits of usability testing
By systematically observing how real users interact with your site, usability testing can help you:
- Identify pain points and flaws in your design
- Improve user flow and task efficiency
- Validate whether users understand your design
- Discover unexpected user behaviors and unmet needs
In spite of what people may ask you to discover, on its own usability testing can’t help you:
- Measure overall user satisfaction
- Test for demand or product market fit
- Find better performing designs
- Evaluate user preferences
Now usability testing can play a role in helping you increase user satisfaction, building a more usable design, and much more. But you can’t go into usability testing expecting to get data on user preferences or expecting a more perfect design to emerge — usability highlights issues for fixing, it can’t tell you how to fit them.
The heart of the problem addressed by usability testing is that business owners and UX designers have to make assumptions about how users will interact. They can make educated guesses, based on the best practices for website navigation, web design, typography elements, user psychology, and their own personal experience from designing in other contexts.
No matter their level of talent or experience, designers have to rely on assumptions, and every assumption builds risk into the design.
What usability testing does is provide real data on actual people using the product. Instead of working from assumptions and guesses, designers can base their decisions on objective, observable results.
Is user testing the same thing?
It depends. You may see the terms “usability testing” and “user testing” used interchangeably. You might also see people who use discrete definitions for both terms.
In this post, I am only going to refer to usability testing because that’s what it is. It’s not a test of users, it’s a test of the design’s usability. The participants in the test who play the role of users are not being tested — they are the ones testing how efficient, intuitive, and easy to learn your design is, and where it falls short.
My take: You can use whatever term you want, just be consistent about it.
When is usability testing valuable?
Usability testing usually starts with a clear business concern: revenue is dropping, users are annoyed, churn is increasing, or key tasks take too long.
In these cases, the goal is straightforward — identify why the design isn’t working and what needs to be changed in order to fix the problem.
But you don’t have to wait for a problem to get started. Regular usability testing is valuable, and many companies use it proactively at key moments, such as:
- During prototyping and design: Testing early helps catch usability issues before they become costly to fix. You can validate design concepts and ensure user expectations align with functionality.
- Pre-launch: Running usability tests before rolling out a new product, feature, or update is a good way to reduce the risk of shipping something confusing or broken
- Before and after a redesign: Testing before a redesign quantifies existing usability pain points. Testing after ensures your changes actually make things better—not worse.
- Annual benchmarking: Companies run usability testing annually (or more often for complex sites and apps) to track whether design changes improve or degrade the user experience.
- Comparing different designs: You can test different versions of your own site, variations of a product, or even compare against competitor apps to identify usability gaps and opportunities.
Another way to think about when to employ usability testing is to think about it as part of the development process. In the words of Garry Tan, usability testing helps you “figure out what’s broken before you build,” and forms an essential part of the product development lifecycle.
As the CEO of Y Combinator, the longstanding silicon valley incubator that has funded some of the world’s most successful startups, Tan understands better than most the risks of neglecting usability testing.
Usability testing formats
This is a quick overview of your options for structuring a usability test. Each of the three formats comes with clear benefits and costs.
In-Person Monitored: This is live, “in-lab” usability testing where a facilitator sits near the participant, observing their interactions and potentially asking follow-up questions. Participants may be encouraged to think aloud while engaging with the product. Post-task questions, surveys, or post-test interviews may be included — or the test may be designed to be more hands’ off, with fewer interruptions.
- Benefit: Provides the richest, highest-quality data. The facilitator can observe body language, catch subtle hesitations, and dig into the “why” behind user behaviors in real-time.
- Cost: Most expensive, even as a DIY effort. Recruiting the right participants, setting up a controlled testing environment, hiring trained facilitators, and using testing software all add up. Many companies outsource in-person usability testing to vendors with established recruiting pipelines, controlled locations, and trained facilitators.
Remote Monitored: This is a synchronous (real-time) usability test conducted remotely via screen-sharing and video conferencing software. A facilitator observes the participant’s interactions, asks follow-up questions, and may request them to think aloud while completing tasks.
- Benefit: It offers many of the same benefits as in-person testing, such as real-time probing into usability issues, but without the challenges of setting up a physical testing space or the geographic limitations of recruiting participants.
- Cost: More affordable than in-person testing but still requires scheduling, trained facilitators, and reliable remote testing tools. Companies may outsource the entire testing process, or some of the responsibilities such as recruiting participants.
Remote Unmonitored: This is an asynchronous usability test where participants complete assigned tasks on their own, typically on a specialized usability testing platform that records their interactions. Along with calculating metrics like task completion rate, platforms collect feedback through automated surveys, session recordings, and heatmaps.
- Benefit: Compared to other formats, this is easier and faster to scale, which is important if you are interested in statistically significant results. It enables quick feedback from a broad audience without the need for real-time facilitation.
- Cost: Significantly less expensive than other formats, but the real cost is the lack of real-time follow-up questions, which puts a low ceiling on the potential for qualitative data. Without a facilitator, it’s harder to interpret why users struggle with certain tasks.
Usability testing terminology
You should understand and be able to explain all of the terminology involved with usability testing. What is a user task? What is a user need? And how — precisely — are these concepts different?
If you have any sort of mushy conception about usability testing terminology, it will cause tons of problems for your team (who won’t be speaking the same language) and during meetings where you will struggle to explain your rationale to stakeholders.
Below, I’ve laid definitions for key usability testing terminology. I’m not going to claim these definitions are perfect, but they are logically consistent and aligned with accepted UX best practices.
Let’s start with terminology specific to usability testing:
- Goal: This is what the user wants to accomplish, the outcome they want by the end of the process.
- Intended path: The designed course of action necessary to reach the goal.
- Needs: These are requirements necessary to get to the goal.
- Tasks: These are behaviors necessary to get to the goal.
- Pain points: These are obstacles and points of friction that block a user from completing tasks, need to be overcome, or stop the user from completing a task.
- Errors: These are mistakes or deviations from the intended path of completing the task.
Let’s take a look at an example that uses this terminology:
A facilitator observes a participant try to accomplish the goal of paying a bill online. The participant needs include their credit card info and login credentials. They must complete several tasks, such as entering their account number, authenticating their identity, creating a new payment, and entering their credit card information. Pain points, such as a long payment form with unnecessary fields result in errors that require the participant to re-enter the same information multiple times.
Now let’s look at terminology specific to the analysis of usability testing:
- Observations: User behaviors you can see but don’t yet understand. You know what users did, but not why they did it, what they were trying to accomplish, or how they felt.
- Findings: Patterns or facts uncovered during testing that have some explanation. For example, 5 of 7 users couldn’t find the bill pay button, or users explicitly said they were frustrated searching for it.
- Insights: Findings with clear causes and implications. For example, users can’t find the bill pay button because it blends in with other elements, leading to frustration and abandoned payments.
- Motivations: The underlying reasons driving user actions. For example, avoiding late fees may motivate someone to pay bills online. Motivations aren’t always conscious.
If you are running remote, unmonitored usability testing, you are not likely to get insights. How could you? You simply aren’t getting the kind of qualitative usability testing data you need to understand the causes of user errors that you observed.
The reason I bring this up is that many people (especially those that are unfamiliar with UX) want to treat observations and findings as if they are insights. They are not, and if you are the person who owns usability testing, you are going to have to police this.
For example, someone comes in and says “We saw users do X, we think it’s because of Y, and so we’re going to do Z.”
Okay — what are the assumptions here? Why are you sure that it’s because of Y and not P or Q or anything else? How confident are you that X is because of Y?
Really push people to show their work if they are proposing fixes for the usability issues uncovered during testing. Don’t let them pass off findings as insights and sneak assumptions into your design that add to the risk of frustrating users instead of mitigating it.
There are ways to supplement that data you get during usability testing that can help you turn observations into the insights you need to be confident making changes to your design. We’ll turn to those next.
Ways to augment data during usability testing
If you’re going through the effort of recruiting and compensating participants, adding these supplemental methods can surface valuable insights, sometimes with minimal extra effort.
Here are some of the popular methods that researchers gather richer data during usability testing:
- System Usability Scale (SUS): A standardized 10-question survey that provides a quick, reliable measure of perceived usability. There’s tons of SUS data out there, making the survey useful for benchmarking and comparing designs.
- Single Ease Question (SEQ): A simple, one-question survey that asks users to rate task difficulty immediately after completing a task. This is good for identifying frustration points without interrupting the test flow.
- NASA Task Load Index (TLI): This is a time-consuming survey that assesses the perceived workload of each task across six dimensions, such as mental demand, effort, and frustration level. It is widely used for evaluating complex tasks.
- Eye tracking:Specialized software or hardware that tracks exactly where users look on a screen. It helps researchers understand visual attention patterns, how users scan, whether they notice key elements, and so on.
- Post-test interviews: Typically short, structured conversations after usability tests to gain additional feedback on user decisions, frustrations, expectations, and motivations. Long user interviews can also be extremely useful, though they are not the norm.
- A/B testing: This is not a usability testing method per se, but running A/B testing can complement usability test findings by figuring out which design variations lead to better performance in real-world usage.
- Heatmaps and click tracking: Tools in this category create visualizations from analytics data that show where users engaged, clicked, and scrolled. Heatmaps are helpful for validating findings from usability testing and comparing performance between versions of a product or site.
Not all of these supplemental methods will be valuable. It depends on your goals for testing. That said, it’s hard to see how implementing surveys to gather qualitative data would be unhelpful in making decisions about how to improve the design.
How To Run Usability Testing
Usability testing is not glamorous, but this process gets results — better user experiences and fewer frustrations. Here are the essential steps:
Even in its most simple and streamlined forms, usability testing is a lot of work to do properly. And there is zero, absolutely zero value in shortcutting any of these steps. It will skew the data in a direction that cannot possibly be accounted for.
For example, if your test participants are not drawn from your target audience, then you can’t really have any confidence that the test results will hold true in the open market. If I am selling enterprise payroll software, but we recruit participants who are self-employed small business owners, then there is just nothing solid to infer from the results.
Similarly, if you are trying to gain insights about what your users are motivated by as they interact with your site, then you have to build a way into your test to get that data. It’s going to be more expensive, time-consuming, require trained facilitators, and likely require a battery of questionnaires, user interviews, and additional research methods.
Tips for Facilitating Usability Testing
When facilitating a usability test, the stakes are high — not just for the quality of your results but also for the integrity of the process itself. Here are some essential tips help you stay on track while you conduct the actual tests:
- Always obtain clear, documented consent before beginning any usability test. This is a big one. Provide participants with an informed consent form outlining the test’s purpose, recording methods, and data usage, and ensure verbal confirmation before starting the recording.
- Use clear and neutral instructions. It’s important that the task or tasks assigned to the participants are not open to interpretation.
- If you’re testing in-person or viewing the participant on a webcam, keep an eye out for visual cues. For example, a user might not vocally say, “I’m frustrated,” but they could furrow their brow or make another gesture to convey frustration.
- Avoid leading participants during the test. Let them interact naturally with the system while you observe. If they get stuck, gently prompt them, but allow for a natural flow of feedback. Observing their struggles can provide valuable insights into areas that need improvement.
- Keep an even tone. Assuming the moderator and participant are having a two-way communication in real-time, the moderator shouldn’t agree or disagree without the user’s comments.
- Remain silent as much as possible. The best usability tests mirror real-world scenarios. In real life, a website visitor wouldn’t be fielding questions about their experience during a session. That’s why tests that just record user behavior are so beneficial.
- Have testers use real money. If you’re asking someone to make a purchase on your ecommerce site, have them complete the actual purpose process, using their own credit card. Obviously, you’ll reimburse them, plus compensate them for the time. But using real money is the best way to emulate a real-life scenario. Participants won’t just rush and pick the first product they see; they’ll actually shop around.
Should we outsource usability testing?
One takeaway that I hope at least a few of my readers have is that they should outsource usability testing to a third-party.
If you have the resources to run proper usability testing in-house, go for it, by all means.
But if you don’t have the resources in-house, there is a lot to be said for hiring experts to handle this. These companies have built recruiting pipelines for participants, which means you can get the ideal users for your tests far quicker than you can on your own. They also have trained facilitators on staff, relentlessly tested processes and surveys, and sometimes decades of experience running usability testing.
I’m not saying that DIY is impossible, but you are going to have to spend a lot of time and money creating the conditions that make usability testing results valid and reliable.
