-- from phone prompts to an auto-filled blog post
From Sep 6 to Dec 17 at Carnegie Mellon University, Pittsburgh, PA
Yulin Liu | Rui (Karen) Ning | Elizabeth Onstwedder | Lena Wang
MakerEd Movement | Documentation | User research | Field research | Prototype | Iteration
PART I: OVERVIEW
A group of Quaker Valley High School students is participating in an internship collaboration with CMU’s ETC. Students don’t currently have much experience with documentation, in particular, with the compose stage of the process. We see that students do capture, manage, and reflect to some extent, but do not compose at all. Therefore, our design focus will be on the specific problem of scaffolding the composition of documentation, specifically, in such a way that students show evidence of their learning and their reflections on the experience.
We believe that by scaffolding the portfolio-building process for QV students during their ETC internship through multiple checkpoints, we can help them see the value of documentation, and articulate and reflect on their learning through documentation. We will know we have achieved this when students produce documentation that identifies critical turning points in the project, and add to/update their documentation regularly.
Our Design Concept
Our design concept consists of two parts: phone prompts (to scaffold reflection), and an online blog post platform with blog post templates (to scaffold composition). The student would be able to customize how frequently to receive phone prompts on the online platform. When the student goes to the online platform and selects a template, the template is auto-filled with the responses to the phone prompts, to further scaffold the composition process.
PART II. RESEARCH PROCESS
1. Literature review
2. Open Portfolio Research
3. Observed two workshops
4. Stakeholder interviews
5. Student survey
6. Student interviews
Part II-1. Literature Review
We did some literature review on portfolio practice and design. We’ve referred two important readings as guide in our design of portfolio documentation process and prototype.
Part II-2. Open Portfolio Research
According to Darren Cambridge, there are 5 stages in the portfolio process: capture, manage, reflect, compose, and analyze. We used the 5-stage process to design our portfolio experience map. We eventually chose the compose stage as our focus.
Part II-3. Observation
We observed two workshops:
● Design workshop at ETC (Sep 27, 2017)
● Project pitches at Quaker Valley High School (Oct 26, 2017)
Research: Observation During the workshops, we focused on observing students’ interaction with each other and with ETC instructor John Balash. We later used our notes to help design our student survey and interview questions.
Part II-4. Stakeholder Interview
We interviewed two stakeholders from Quaker Valley and ETC to understand their perspectives on portfolio documentation in this internship, as well as their expectations for student learning. We created user profile boards to summarize our findings from the interviews with them.
Part II-5. Student Survey
We sent out a survey to the students through Amy to learn about students’ backgrounds, experience, and attitudes and opinions about documentation. Twenty-two students are participating in this year’s ETC internship program and 17 responded to our survey. Here are two findings that we want to highlight:
1. Many students (100%) consider concept sketches important to save. Over 80% consider prototypes and feedback important to save, and 70.6% think team notes and presentation are important to save. Personal notes are considered least important to save (58.8% of students said it was important to save them). It is encouraging that more than 50% of respondents felt it was important to save some evidence of their project process.
2. A surprisingly high 94.1% of students like to share work with friends. 64.7% like to share with family, and 33% like to share their work with either teachers or professionals.
Part II-6. Student interviews
Based on the survey findings, we interviewed 7 students who met the following criteria:
Both students who participated in the ETC internship last year, and students who are new to the internship this year
- Students with strong backgrounds in different areas, particularly computer programming, and art & design.
We used affinity diagrams to synthesize our findings and later on translated the Problems and Needs identified in students’ portfolio practice into insights.
We found from these interviews that:
● Students do save a lot of artifacts (code, prototypes, pictures etc.) from the design process. Some students save different versions of their work, but do not formalize reflections such as by writing them down.
● Some students do have a clear goal regarding documentation of their project work and using it as showcase to assist their future college application. Even with a clear goal, it seems they don’t know which platform or tools to use or how to convey it to show their learning experience.
Below are two personas we made to synthesize our interview findings:
Some students like to share their work with others, but are concerned about receiving negative comments from people who don’t know what type of work is involved in the projects. This shows students are looking for like-minded people to share their work/experience with.
Based on our research, we built a documentation process model and used it to examine what students actually do with the portfolio practice. The missing parts led to our design opportunity.
After taking those deeper research by starting the survey and interviewing diverse students and stakeholders, we decided to develop a responsive-web system which can let students documenting their design process by writing blogs, we decided to combine phone prompt and computer-platform blog writing together.
PART III. IDEATION
We used storyboards and scenarios as brainstorming techniques to create a first version of our design and then created a portfolio experience map to consider the start to end experience of using our design concept.
We presented the map to our stakeholder, Amy Keller, and learned from her thoughts about our design concept. She suggested we particularly focus on finding ways for students to have options, which she identified as important in maintaining students’ motivation to document. We also consulted our professor and TA to improve the design of this map to convey a clearer concept. Here is an edited version of the Portfolio Experience Map:
PART IV: PROTOTYPING & USER TESTING
We decided to use a paper prototype because a simple prototype can be an effective way to investigate the role of a design concept, and the low fidelity encourages users to share their thoughts and feelings honestly
USER TESTING 1: PROCEDURE
Step 1. We gave the student the cardboard phone with the first phone prompt, and asked them to use the prototype as if it was a regular phone. After they pressed a ‘button’, we replaced the current screen with the appropriate next screen.
Step 2. We asked the student to handwrite their replies to the phone prompts on small pieces of paper.
Step 3. When they finished the phone prompt and chose a blog template, we put the pieces of paper where they had written their responses to the prompts on top of the blog template to simulate the autofill feature.
Note: Two team members worked with one user at once. One team member facilitated the student’s interaction with the prototype, and another recorded the session and took notes. We tested our first prototype with 4 students.
USER TESTING 1: RESULTS
Students liked the simplicity of phone prompts, the auto-filling feature, and the templates. But we still weren’t sure if students would really use our design throughout the semester:
“I might use it a few times and then stop.”
We decided to add two features to our design:
- Customize phone prompt frequency. We hoped that by making it possible for students to choose how often they receive the phone prompts, they could choose timing that made it more likely they would continue to use the design.
- Sharing feature. We hoped that being able to share their blog posts and get feedback on their projects might help show students the value of documenting, and increase their motivation to document using the design.
USER TESTING 2: PROCEDURE
Goal of prototype 2: See whether the changes made students more likely to use our concept .
We decided to use an interactive digital prototype for our second round of prototyping, for a more realistic experience while testing. We created the prototype for the phone prompts using Sketch and InVision, and the prototype for the blog platform using Google slides.
Step 1. We asked the student to select how frequently they would like to receive phone prompts.
Step 2. We gave the student one of our phones and asked them to proceed through the phone prompt. We asked them to say aloud what they would type into the prompt on a fully implemented version of our concept.
Step 3. We asked students to click through the blog platform prototype on one of our laptops. Again, we asked them to say aloud what they would really type.
USER TESTING 2: RESULTS
We had two main findings from this round of prototyping:
- Students with more expertise in the types of projects required for the ETC project wanted more freedom, particularly in writing the blog posts.
- Students were not interested in sharing their work on social media. They mostly wanted to share with teammates, and occasionally with instructors. As a result, we decided to make two changes to our prototype: 1) Add a blank template option, and 2) Remove social media sharing options.
PART V: FEEDBACK & REFLECTION
Critiques from reviewers
One reviewer noted that students setting up how often to use the phone prompts might result in some students choosing to do it as little as possible. Someone in the Q&A also suggested requiring students to complete the prompts once a week at the beginning of the internship, and then giving them more choice later. We think this is a good suggestion, and would incorporate it into our recommendations to Quaker Valley.
Another reviewer suggested that we may want to explore more deeply students’ reason for not wanting to share their learning. We actually did look into this in our research, but were unable to address it in a relatively short presentation. We think a supportive classroom environment (as we are sure Quaker Valley has!) could help, and anticipate students becoming more comfortable sharing their work throughout the semester as they get to know each other.
- Somebody noted that phone prompts may occur at inconvenient times, and that it might make sense to have them occur after work periods - we agree! However, there is no set class time for this internship. Perhaps internship facilitators could encourage students to set up the prompts to occur after their group meetings.
- Some reviewers suggested things that we thought were excellent ideas for future iterations, but did not have time to add in this version:
- ○ Adding a feature to address effectiveness of team as a whole
- ○ Adding more detail on how a feedback mechanism would work
My final thoughts...
Thinking back at the “wicked problem” reading at the beginning of the semester, I have now come to understand what an important lesson the reading was trying to tell us. By experiencing the whole process of researching, designing, presenting, and iterating, I have learned a lot more about what it means to be designing.
Our Team Process: