Turning "Inappropriate" Questions in Usability Testing to Meaningful Ones

A few years ago, I went to a UX training day-long conference on usability testing led by Nielsen Norman Group (NN/g), the leading research & consulting group in the UX industry. This month, I was luckily invited to do an intro-level training at my company on usability testing for some research assistants who are interested in the UX world. I decided to review my notes from the NN/g training.

The training would be an overview of UX interviews and usability testing. There is one part that covers interview techniques. From my NN/g conference notes, I found a few questions that were “inappropriate to ask” during interviews as a UX researcher. I decided to use them as examples to walk through, while I offer suggestions to improve the questions. I’d like to share my writing and thoughts here.

1. “What do you think of the colors and fonts?”

Check design standards when designing; use other tasks to find out if the colors and fonts work, like a comparison.

2. “Would you use this feature?”

Make it more open-ended and more neutral. E.g., “How do you see yourself use this feature? If you don’t want to use this feature, why? 

3. “Did you think it was easy to use this website?”

Put it on a scale and ask it more neutrally, “How easy or difficult was it to use this website?” Showing a scale with labels at various points would serve as a supportive visual reference too.

4. “Did you pay attention to these images?”

Well by asking this, the participant will notice it now. Also, it sounds like an interrogation. If you think the participant didn’t pay attention or see the images, ask them what might have happened, or what did they see on the page. Start from there.

5. “It seemed like you were having difficulty in this area. What were you thinking?”

“Area” is a big word, make it more specific. Turn the question into “it seems like it’s not doing what you want it to. What do you expect to happen here?” Do it in an inquisitive way, not a judgmental way. It’s not the user’s fault.  

6. “Why did you not scroll down to see all of the options?”

Give a hint first: by asking the participant “do you think this is everything on this page,” the participant could start to try different actions to see what else is possible of viewing. Then, observe if the participant is familiar with the scrolling action. (It might not be as common as we thought.) Before this usability test, make sure to conduct a heuristic evaluation to checking the affordances of scrolling (e.g., cut-off design). 

7. “Did you notice this link?”

Well, now they cannot not notice it. It’s leading and not helpful in gaining new information. Use other tasks to find out what might be the issue.