Triangulating Users' Self-Evaluation and Task Performance Data to Understand Audio Quality
CHALLENGE
Imagine that you’re on the phone with your friend who has their (Bose) noise-canceling headphones on, but someone else is talking loudly next to them. They might not be able to hear the noise around them (because of the noise-canceling headphones), but what would your experience be? Would you hear the noises around your friend? With the NC 700 headphones designed and produced, Bose wanted to understand the expected improvement of these headphones’ voice pick-up functionality compared to its previous generation and other companies’ products. While studies have been conducted from the engineering side, Bose wanted to understand the users’ perspective.
TEAM
The UX research team of three consisted of a lead user researcher specializing in linguistics research and quantitative data analyses and two user research co-ops, I being one of them. I was in charge of the study setup (arranging a total of 16 unique playlists for all the participants to listen to and transcribe), running all hour-long user research sessions, transcribing participants’ written files, and compiling a report draft for the lead user researcher to review and finalize data analyses.
RESEARCH APPROACH
The methodology is a mix of Wizard of Oz, survey questions, and task analyses. Participants were divided into four groups by headphone product model, each group using only the stimuli recorded with the respective pair of headphone devices. Stimuli were sentences pre-recorded in four different noise environments and were played to participants via the same type of noise-canceling headphones—to block surrounding noises. All participants were blind to which group they were in. Participants were asked to perform certain tasks regarding the stimuli they heard via the headphones.
Analogous photo: each participant navigates in the company while completing tasks wearing headphones.
RECRUITMENT
All participants (16) were external participants selected from the Bose User Research panel who met a list of requirements on their physical conditions.
PROTOCOL
Researchers would guide each participant through a tour of the company, where at various points, play the participant a series of audios recorded in different environments by one of the four headphone devices. Each headphone device was also used in four different contexts with different levels of background noise to test the products’ performances in different noise contexts. Between checkpoints, researchers also use Likert scale questions to participants about the intelligibility and acceptability of audio.
ANALYSIS
We analyzed data in three main categories,
Functional Intelligibility (participants’ performance on tasks) across contexts and devices
Perceived Intelligibility (participants’ perception of the task's easiness) across contexts and devices
Acceptability (participants’ level of acceptance of the audio quality) across contexts and devices
We also looked into where there were correlations between the categories above.
INSIGHTS & OUTCOMES
One major finding from the study shows that NC Headphones 700 perform significantly better in the context where someone is sitting next to the headphone user and speaking. In another word, the NC Headphones “picks up and isolates your voice while canceling the noise around you.“ For more information, please refer to the Bose Product page.
PERSONAL REFLECTIONS
Through this research study, I was able to practice and develop my skills in facilitating research. I learned to juggle many balls while appearing cool and guiding the participant to any next steps. Before each session, I would make a personalized audio playlist consisting of about a hundred audio clips ordered in a specific fashion. During the session, I take a participant on many “campus tours” while coding their behaviors. Not having a naturally good sense of direction in me, I had to make sure I don’t get lost as the guide. (At some point, one of the participants was more interested in the company interior design than the tasks they were asked to do. I had to control the situation, but I learned to be “never rude to a participant.”)
I wrote a piece on what I learned from this experience facilitating the sessions, “YOU Take Control as a User Researcher.”
