Leading Product Research From 0-1
Explorative and Evaluative Methods to Study Users’ Website-Building Experience
THE TASK
If you’ve ever explored building a website to showcase your work or sell something, you may have heard the term “website template.” How much work do you think the template does for you, and how much do you think comes from you outside of that template?
From earlier research, we learned that some users had a hard time picking a template, and they would give up website building altogether. How might we help these users start with a template that meets their needs? We then set a goal to design a guided experience for the users to help them create a custom template.
TEAM
I was embedded in the product team as their UX research partner. My cross-functional teammates included a product manager (PM), 1 technical product manager (TPM), 2 product designers, 1 content strategist, and 3 engineers. I also collaborated with data analytics and marketing partners regularly.
During each round of research, I collaborated closely with PM and designers on aligning goals, research questions, and stimuli. I also worked with the data analysis partner to make sure that my (qualitative) research could dive into understanding the “whys” behind outstanding numbers from their work.
I supported this product team from 0 to launch. Multiple rounds of user research, foundational and evaluative sprints were conducted to inform the product iteration.
RESEARCH APPROACH
RESEARCH ACTIVITIES
I kick-started this 0-to-1 project with foundational research to understand users’ needs and pain points. Then, I tested individual (competing) concepts that designers put together to check users’ perceived value of the innovative ideas. Later, a second round of concept tests was conducted on the flow of the multi-step product. Most recently, I led a usability test of the live MVP experience, in which insights were shared to guide the next steps of iteration.
Main methods since project kick-off:
FOUNDATIONAL RESEARCH
CONCEPT TEST (ROUND 1): INDIVIDUAL CONCEPTS
CONCEPT TEST (ROUND 2): FLOW
PROTOTYPE WALKTHROUGH WITH (NEW) TARGET USERS
TARGET USERS’ BEST PRACTICES: USAGE ANALYSES
MVP USABILITY TEST
RECRUITMENT
n=16 for foundational research
n=8 for each round of concept test
n=24 for usage analyses
n=16 for usability test
Participant criteria: those who fit the general profile that Growth org set out to solve for (e.g., interested in building a website, don’t have a website yet)
Process: I wrote up screening questions (multiple choice questions and open-end questions) and sent them out via vendor platforms. I collected the responses (usually 200+) and selected participants that best fit our detailed criteria. I finalized the participant list with my teams to make sure the cohorts were accurate and aligned.
PROTOCOL
In each round of research, visual stimuli were included because website building is a highly behavioral and visual activity. In the foundational research, 1:1 in-depth interviews were the main method to dive into participants’ past experience and perception of website building; still, two tasks were embedded in each session for participants to build something. Their behaviors, opinions, emotions, and questions were captured by me and my stakeholders who were observing and taking notes for the sessions. My discussion guide was designed to elicit specific feedback to inform the product team’s decision-making.
The research plan always made sure to include opportunities for participants to work on certain authentic tasks as well as share their attitudes. The research activities were robust enough so that the feedback wouldn’t be too vague to inform the Product team’s decision-making.
ANALYSIS
After each round of user research, I always debriefed with stakeholders to capture their top takeaways, surprises, and questions. Then I started putting together a topline document (for quick turnaround) followed by a deck for cross-team share-out.
INSIGHTS & OUTCOMES
In each round of research, I synthesized data to answer the main research questions. In foundational research, we gathered insights on (1) users’ challenge areas and (2) the main principles of a guided experience. These main principles continued to be guiding pillars for future concept and feature design.
In the concept test of individual concepts, we captured insights into users’ impressions of innovative but competing ideas (e.g., Did users find more value in a concept that’d give them more support or more control? Did users find more value in a concept that’d give them more open placeholders or one that showed more progress but was irrelevant to their line of work?)
In the concept test of the prototype flow, the learnings helped the team understand how the design elements positively/negatively impacted users’ confidence in launching the site.
EXAMPLE FINDINGS: FOUNDATIONALUsers want transparency on what parts of the custom-website-building process helped put together the template in the end.
Users desire unique designs that help them stand out and represent their brand.
Users don’t want to be boxed in.
EXAMPLE FINDINGS: EVALUATIVE Some participants missed out on fully customizing Step 1 section-by-section.
Some participants didn’t know there was more than one section to interact with.
It’s not clear that multiple selections are supported and desired on this page.
[X number of] participants saw multiple sections but only interacted with one before they clicked “Next.”
One participant clicked “Next” by mistake. Wanting to return to this page and continue editing, they failed to make their way back.
Participants are not registering the context shift from Step 3 to Step 4.
The transition is jarring to some participants.
Language and repetitive content titles confuse participants.
Users don’t readily understand or differentiate various website components the way we do, which brings challenges to these two steps as a whole.
