Open-Box and Onboarding Experience for Voice-Enhancing Headphones: How Do Users Follow (or…Not Follow) Instructions?
CHALLENGE
The team at Bose was developing a new conversation-enhancing headphone device. An (onboarding) app was designed to walk through users how to wear the devices and control various settings.
Bose Hearphones are an analogous product to the project I was working on
Analogous illustration to an in-app illustration
Analogous photo from internet: by Valcarcel/WIRED
TEAM
The team consisted of one app UX designer, one package designer, and two user researchers. I was the user research co-op who worked with the lead senior researcher in setting up the lab and taking notes for the interviews. We later collaborated on writing a comprehensive formative study report for the FDA to review.
RESEARCH APPROACH
The primary goal of the study was to evaluate the effectiveness of the onboarding of the new headphones to identify strengths and opportunities for improvement, which included the mobile app, hardware accessories, and documentation.
RECRUITMENT
We recruited 8 external participants from our Bose User Research Panel. All of them were individuals of 18+ years of age with a certain perceived hearing condition.
STUDY SETUP
The lead researcher sat next to the participant. A lab iPhone with the onboarding app loaded was connected to Mr. Tappy streaming kit (for recording mobile devices from a user’s point of view, without getting in their way) so that as a note-taker, I was able to see from my monitor not only what happened on the phone screen but also where the participant was tapping (how their hand was interacting with the device).
Study materials included a sealed product package, an iPhone with the onboarding app, and also supplies like a hair band, handheld mirror, and hand sanitizer for the participant to freely interact with.
PROTOCOL
Each study session lasted up to 1.5 hours, during which researchers asked the participant background questions, administered use scenarios, and conducted a final interview. During use scenarios, participants were asked to think aloud and respond to researchers’ follow-up questions. Likert-scale rating questions were also asked at different points of the study session, e.g., “how confident were you in putting on the headphones correctly….”
Background interview
Use Scenario 1: Onboarding: Donning the headphones
Use Scenario 2: Eartip Replacement
Use Scenario 3: Locate help information
Knowledge Tasks
Final Interview
INSIGHTS & OUTCOMES
Researchers held informal meetings to share top findings and recommendations with the project team before coming up with a comprehensive report. The findings were responding to the study goals. Below are a few examples:
Study Goal 1 (example): Assess the effectiveness of onboarding in guiding users in ear-tip sizing, fitment, removal, and replacement.
Finding: Participants successfully replaced and removed ear tips; the majority did so without the application.
Recommendation (example): we need to focus on ensuring that participants know multiple ear-tip sizes are available via multiple methods.
Study Goal 2 (example): Collect users’ subjective feedback about the onboarding / out-of-box experience.
Finding (example): Most users described they would complete onboarding at least up until the Feature section’s end; however, several participants did not initially realize they needed to download an app to set up and use the product.
Eight recommendations were provided regarding packaging, labeling, and indications at different places to download the app and finding additional ear tips.
PERSONAL REFLECTIONS
Through this experience, I learned the importance of testing early and often, as well as constantly asking the right questions. I was exposed to the thoroughness of a formative study at an early stage of product development. There they are, a hardware prototype and an app — how might we use the prototype to find out how users are able to use them as expected like they were doing it at home? If participants failed to follow the onboarding steps we wanted them to follow, could they still complete the tasks? If not, why? I have been observing my lead researcher build rapport with each participant, make small talk with them, and challenge them with “tell-me-mores” and intimidating probes like “you seem hesitant, tell me more.” Even when the participant said “I like it,” she’d follow up with a “why.” This whole process of getting to the bottom was eye-opening for me.
Working on this project taught me how to become a more effective notetaker. I got into the project as a confident note-taker (I was able to memorize long sentences on the fly while typing at an impressive speed). I thought that if I really tried, I could take note of everything and fill in everything in the designated cells, including participant behaviors, researchers’ questions, and participants’ think-aloud content. It turned out that it wasn’t realistic—these “raw talents” of mine—harmless but not enough. I had to understand that note-taking was a skill under development. I used to think that recordings and video-taping were for slouchy researchers and note-takers: “nobody would have the time to revisit them,” I thought. It didn’t mean the recordings were not important though. For a project as high stakes as this one, to fill in the gaps for my note, I had to watch the same clip over and over again to take a closer look at their interactions with the hardware prototype as well as the software/app. Sometimes I wish for a better angle from the camcorder that could take in a better view; there were other times when I wish I had asked a follow-up question to make sure that I got everything down. One tool I started to use often as a note-taking tool thanks to my project lead was a horizontal excel spreadsheet. The columns were titled by expected user behaviors and steps (open package seal, take out headphones, read manual, etc…), and the rows were by-participant. Still, with a good tool, I still had to focus during the session and resort to the videos to fill in the gaps afterward, but the tool would help me stay focused and oriented at the moment.
This experience also taught me how to analyze the different pieces of the results and tie them all together. I learned the importance of staying objective throughout a research project; although I had an idea of the results from the research, I had to lay them out and look at them to be objective. Writing a report was part of this process that helped me stay objective. I didn’t know that despite all the in-depth interview sessions with the participants, the bigger part of the study (precisely, the part that takes longer time) did not start before working on the report writing. My mentor took the time to teach me how to put facts into words concisely and pertinently. For a formal report (well, for the FDA) or for writing in general, I was challenged to think about how to lay out the facts, check each of the study goals we originally listed in the protocol, and make reasonable recommendations based on the facts explained. To achieve that, during the fact-laying part, I had to be patient in digging into the data, aka the video recording in this case. For example, one time a participant used the handheld mirror to check whether they put in the headphone ear-tips correctly (as the app instructed on one screen), and I don’t have a column in my note-taking sheet that records the data of whether a participant checked the mirror or not. I wouldn’t know whether most of the participants were willing to follow the instructions and check, or why they would or wouldn’t do that. We had to find out and collect those data. In the end, we found in the videos that three participants checked with the mirror, as well as the reasons why the other participants didn’t check.
