Getting started with usability testing
My scrappy start may hold some helpful clues
Some years back, I was working as a web designer at a large company when I realized the power of usability testing.
Until this time, I’d typically design something and show it to coworkers for approval. Fellow designers, stakeholders, directors, would weigh in with often contradictory opinions.
But the UX design field was rapidly maturing, and the benefits of user research were becoming harder to ignore.
It was time to learn how to conduct usability testing, and it would change the entire conversation.
Getting it going
Lean UX by Gothelf and Seiden, and Don’t Make Me Think by Krug, kept me company on the New York subway.
Don’t Make Me Think was embarassingly easy to read, filled with practical advice for open-minded beginners.
At work we had ongoing design work to test, and our in-house account managers knew the end users extremely well. For the moment, they would serve as my starting point.
Prior to the test
Designers on the team already understood the craft of designing things well. This is important.
It’s harder to get going if you haven’t mastered composition, if you don’t know design patterns very well, or the design and prototyping tools, or if you don’t understand what the engineers can actually do.
In this instance, I worked with another designer to determine the typical things a user would want to do while using the software.
While my teammate designed a clickable prototype using Sketch and InVision, I put together a script that detailed, word for word, what I would say to my participant. Product people were shown this work prior to testing. Now we were ready to recruit.
An actual test
I set a meeting with an account manager who I knew somewhat, and booked a small conference room.
Following the formula from Steve Krug, I started by introducing what this was.
- I reminded her that I was part of the design team, and wanted to learn from her because she was the user expert. I think this made her feel important and more willing to help.
- I said I didn’t design the prototype myself, so if she was critical it wouldn’t hurt my feelings. I paused and she chuckled a little, as I hoped she would.
- I said this was a test of the software, not of her, but she knew that already.
- I noted the software looked real, but that it wasn’t and not everything worked, and not to let it throw her off.
- I asked her if I could record the session (voice and screen capture) for my own notes, and she didn’t mind.
During the test I read from a printed script, which I used to scribble notes. I shared the context; “Imagine you’re a typical user who wants to accomplish XYZ” — using the lingo from the business. “Now use the software to demonstrate how. Think aloud while you do it.”
And she did! And I took notes.
One of the nice things about taking notes by hand is that it signals to your participant that you care about what they are saying. It also gives you something to do, an excuse to look down, making it easier to keep a poker face and not bias the participant.
She may have gone off into other topics, and in future tests with future participants I’d get better at controlling that without shutting down useful talk.
But she shared her thoughts, at least playing the role of the end user, and it was enough to call the operation a success.
More was needed
I knew one test wouldn’t be enough. I think as a beginner I did three of them, though now I’d do no fewer than five per iteration. Some recommend seven.
The point is, you’ll want a series of them in order to look for basic patterns.
I remember watching the recordings of those first interviews, and painstakingly transcribing the notes word-for-word with a lot of pausing and rewinding. I’d later realize this was a waste of time.
You’re better off taking clear notes the first time and simply referencing those and keeping the recordings for other reasons. These may include playing compelling clips to stakeholders, or rewatching genuinely complex or unclear portions of a session.
Pleasant and powerful outcomes
More often than not, I’ve discovered that people are actually happy to sit with you and share their thoughts about your creations. They will often be evaluating something that matters to them, whether they work closely with end users or are the end users themselves.
Often, you won’t even need to pay them for this, though some believe it helps with recruitment.
Presenting findings
It took me some time to figure out the best approach to taking all those notes and making them into something compelling.
These days, I’ll use pen and paper during the usability test itself, and immediately transcribe into a Google doc after the session is over. In moving from ink to digital, I’ll often clarify the language and add necessary context — to make the notes easier for others to understand, as well as for my future self to remember.
When a series of five has been completed, I color code the text by participant and then group by topic. You might want to use some better and more advanced tool, but this is a way that works for me.
Once grouped by topic, it’s much easier to spot patterns.
- “Everyone understood W.”
- “Nobody could find X.”
- “There was no clear winner when it comes to Y.”
- “There was a contradiction when it comes to Z.”
You will also get context by speaking with participants, and that’s especially compelling when talking to end users. They probably don’t work or act like you and knowing about their world and their motivations could be a serious game changer.
I’ll note: problem discovery interviews and other techniques, like field studies, are devoted to these kinds of findings — but these findings can also find their way into your usability tests, and that is something you can embrace.
Once I’ve got my content figured out, I’ll drop it all into a presentation deck with plenty of visuals. Screenshots of the actual UX should be paired as closely as possible to your findings about them. I use shorthand when referring to which participant said what, probably inspired by New York subway icons.
Patterns are called out.
When the multi-slide deck is nearly finished, I’ll take stock in what the learnings are, and construct a summary slide that lists out validations and opportunities.
If you can find a way to present this to an audience of stakeholders, in a forum where you’d normally have their attention anyway, you are likely to win over any internal skeptics.
Keeping it going
One round of five sessions may not be enough. You can go again with five more, after you update your design with any clear findings. At some point you may need a meta-summary slide of all your research for the project.
Recruiting users
When I was just getting started, I tended to work with internal subject matter experts more, to grow as a moderator. I also tend to work internally even now, if the prototype is rough and there are a lot of fundamental questions.
But once you’re feeling good about your approach, you must get your experience in front of real users. Failing to do this will reduce the value of your research, and won’t help you overcome internal groupthink.
Recruitment can have its own costs and challenges, so this is one area where I like to ask for help. Reach out to your account managers, support team, etc., for external clients, or pay for a recruitment service, or leverage social media groups.
Hold your head high
Usability testing helped me redefine who I was as a designer, and made it easier to take on other forms of research.
If you’re ready to take this step in your design career, you may be delighted by the results.
