How To Run Live User Testing, Part 3: The Debrief

Editor’s Note: Brenden Mulligan is a co-founder and designer at Cluster, and previously created Onesheet, MorningPics, ArtistData, and others. This is the final part of a three-part series on how live user testing is run at Cluster in preparation for the next release of their iOS and Android apps.

The first part of this series focused on getting the tests setup, which includes deciding on a specific thing to test, when and where to conduct the user study and what type of users to study, as well as recruiting participants with Craigslist, trimming the candidates list, prioritizing and scheduling candidates, and getting the right equipment.

The second part focused on actually running the tests, which includes arranging the room, meeting the participant, introducing the study, not revealing the answers, simulating app discovery and installation, walking the user through the prototype, and wrapping up.

This post will focus on taking all that amazing feedback you just gathered and parsing it into useful, actionable intelligence.

One more time, I want to give a huge amount of thanks to Michael Margolis and the Google Ventures design team, who taught us most of these techniques.

So what next? You have a set of recordings containing incredible user feedback, and now it’s time to parse and process it so you can turn the insights into action items for your product. Here’s how to pull out the good stuff and get your team on board with next steps:

1. Schedule viewing party
2. Distribute materials
3. Capture insights and ideas
4. Combine everyone’s insights
5. Identify patterns
6. Bucket patterns according to app sections
7. Address the problems & start again

Like the first two parts of the study, this takes time, but it’s very worth it in the end.

Schedule viewing party

The first step is to schedule a time for the company’s key decision-makers to review the videos. Including everyone might sound like a waste, but it’s very important. I know because we didn’t always do this.

When the whole team sits around, you have a much bigger opportunity to understand what feedback is important and what is worth ignoring. You’ll pull a lot more out of the sessions.

Reviewing the videos with the Cluster team

So, even though it seems like a lot of time, schedule time for the core team to watch all the videos in their entirety. Don’t skip this, or you’ve wasted all the time prior.

Distribute materials

While watching the videos, the goal is for the team to capture insights and ideas from what the user is saying and doing. The tools we use to do this are:

Tools for capturing thoughts and ideas

1. 3×5 inch Post-it notes
2. Legal pad
3. Regular pen
4. Thick marker

Capture insights and ideas

We’ve found it helpful to instruct everyone to capture two kinds of things on paper while watching:

1. Important insights (good and bad) learned from the user on the notepad
2. Any specific implementation ideas on Post-it notes

My notes and ideas from 5 videos

The majority of the writing will be insights from the user and should be captured with a normal pen and paper. Examples of insights are things like:

• “Blue button really works”
• “Really understood intro page”
• “loves design”
• “had no idea what to do first”

Everyone should write down everything they notice from what the user is saying/doing.

If the insights lead a team member to have a specific idea of how to fix / improve the app (ex. “Add ‘We will never post’ under the Facebook login button”), have them write each down on their post-it notes using a thick, black marker. Getting these ideas down on paper will let the team member focus on the rest of the study instead of trying to remember their idea.

Combine everyone’s insights

After watching each video, have someone stand at a white board and write down the insights everyone captured. Use a green marker insights for things that worked, a red marker for things that didn’t work, and black for general insights. Only write down things that more than one team member heard. The purpose of this step is to distill everyone’s page of notes into a common list of insights to address later.

Blurred version of the board after one of our debriefs

Create one column of notes for each study participant. Then, somewhere off to the side, have everyone put up their Post-it notes in no particular order.

Identify patterns

Once you’ve watched all the videos, and everyone’s insights are up on the board, it’s time to find patterns. It’s not worth fixing every small issue each person had because all users are different. The goal of the entire exercise is to figure out the re-occurring problems that might also affect users in the wild.

With a new color, circle or draw a dot near any insights that come up more than two times (or more than 40 percent of the time depending on your number of participants). Do this for green, red, and black insights.

Bucket patterns according to the app

Now that you know what common insights came from testing, it’s useful to categorize what part of the app the insights affect. With Cluster, our users were testing four sections:

1. Pre-Registration Screens (including App Store marketing)
2. Creating & Naming Initial Cluster
3. Inviting Friends
4. Uploading Photos & Exploring App

On another whiteboard, we made a column for each section. Then we went through the first whiteboard and moved each recurring pattern over to the appropriate column.

Common insights and patterns

Suddenly, hundreds of insights were distilled and organized according to the part of our app they affected. An actionable list emerged.

We then moved the ideas that had been posted underneath the insights. What we found was that most negative insights already had some great suggestioned ideas for improvements.

Address the problems & start again

Now it’s time to get to work to improve the product to address the common negative insights you found during this process. Then, start over and do the whole thing again. Keep going until you’re satisfied that the problem areas aren’t too big of a deal.

You’ve now completed the entire user testing journey! I know it’s a lot of work, but once you do it a few times, you’ll start to see problem areas disappear and users start to understand your app much faster. It’s invigorating. I’d encourage you to trust the process and see what happens. I guarantee your app will become easier and more delightful to use.