Multi-group Usability Study with Users with Different Experience Levels
How much should a template provide?
CHALLENGE
If you’ve ever explored building a website to showcase your work or sell something, you may have heard the term “website template.” How much work do you think the template does for you, and how much do you think comes from you outside of that template? The Templates team was exploring different template constructs, hoping that one or more would increase users’ trial intent. For example, one concept was a template that gives users more of a blank canvas as a starting point, while the other concept provided some more specific guidance.
Research would come in to help derisk the launch of these constructs. Do these features help users get started with building a website more easily?
We had about 6 weeks to answer these questions.
TEAM
PM, designer, data analytics partner
RESEARCH APPROACH
The research goal was to learn if these website-building constructs/concepts help users get started with building a website more easily. I planned a series of semi-structured in-depth interviews with the following main structure:
Warm up interview
Usability task (building their website): each participant goes through only one of the concepts
task assignment is randomized and balanced
Debrief
I decided to invite two groups of users, experienced website builders and inexperienced/first-time website builders to understand how these constructs may work differently for them, based on our hypothesis that one construct should support inexperienced builders better.
RECRUITMENT
I recruited participants via a vendor platform that the company subscribes to. I set up a screening questionnaire with both multiple-choice and open-ended questions. I screened 250+ responses to finalize a participant shortlist after facilitating the project team’s feedback.
Criteria (part of it):
n=16 (8 for inexperienced, 8 for experienced builders)
(in the end, we had 9 inexperienced and 7 experienced)
Interested in building a new website
PROTOCOL
In each interview session (with one participant), I (as the sole researcher) would conduct a warm-up interview to understand the participant’s experience in building websites and the main challenges that hold them back, which was also a great opportunity to build rapport with the participant. Then, I would direct the participant to the Squarespace templates’ (live) page, where they have 10-15 minutes to work on building their site. Each participant only goes through one of the concepts.
Each session was live-broadcasted (upon participant’s content) to my project team members, who also helped take notes on a shared FigJam board with participants’ quotes and screenshots of their usability tasks. I also provided space for each attendant to jog down their surprises, takeaways, and ideas.
In the meantime, I was managing a live chat thread with my teammates to record any questions they had. I’d follow up with the participant with questions I found to be useful at the time.
ANALYSIS
I hosted an end-of-day debrief session with my project team after every 3-4 sessions so that everyone could “download” their observations and learnings from the day and chat through main themes and topics. I documented the conversations so that I could revisit them and do a deeper dive during my analysis.
Types of data that I analyzed:
Participants’ attitudes: before they land on each concept, first impression of the concept, and their reflection on their experience of it.
Participants’ performance: first actions, blockers, tasks completed.
I also looked into consistencies and differences between the two groups of participants (inexperienced vs. experienced) in the data categories above.
INSIGHTS & OUTCOMES
One major finding from the study showed that people’s expectations of the concepts were different from what their experience is upon editing. Through synthesizing their expectation, we learned how each concept was appealing to them in the first place but how they might bring challenges once being edited.
These findings provided a deeper understanding of (1) users’ expectations and first impression of concepts tested, (2) users’ challenges using different website-editing constructs, and (3) which concept was a better starting point experience for users.
Ultimately, the research findings helped the team understand the main issues of the concepts and reach a decision to pivot to another direction.
