Brainology is a hybrid learning product that helps middle schoolers develop a growth mindset. It consists of an online module and a workbook with classroom lessons and activities.

My Role

I was hired as a graduate research assistant. My role was to work directly in schools collecting qualitative and quantitative behavioral and attitudinal data about users (teachers and students), their use of the product, and their experience with the product.

Specifically, my main role was to work with my direct team of 5 (4 researchers + 1 director) to:
1) organize and execute the implementation of major surveys at three time points (before, directly after, and a semester after the use of the product).
2) collect usability data throughout the year using observation, informal usability tests, interviews, surveys, feedback sessions, workshops, and more.

Other roles I held were to:
- Assist in recruitment for participating schools each summer.
- Lead and iteratively refine training workshops for teachers about how to use the product.
- Collaborate weekly with stakeholders (clients, designers, product owner, developers/tech support, lead researchers).
- Pitch and present redevelopment ideas.

Outside of my duties, using UX insights gathered from the field, I pitched, advocated for, and accomplished the following:
1) Creation of a structured survey administration protocol to organize the internal team.
2) Redevelopment of teacher training slides for workshops to improve learning, confidence, and subsequent lesson quality.
3) A new edition of student workbook (English and Spanish), teacher workbook, and teacher guide to implementation at the end of the efficacy study for future clients to use with greater ease.

Methods of Research

In this complex, longitudinal, mixed-methods research, we used several methods for data collection.
Tools used: Google Docs, Sheets; Qualtrics, Brainology analytics reports.
Ethnographic observation

• The most common method, used weekly to get a feel for the classroom during regular lessons and during lessons using the product. Our team iteratively developed a coding scheme, which we used to align on how to quantify the quality a lesson and student engagement.

Usability Testing

• We observed student and teacher use of the product in their natural setting and conducted unmoderated and moderated tests.

Interviews

• We informally interviewed teachers after lessons and during coaching sessions.

• We formally interviewed teachers and students at the end of the year.

  • I led the script writing, filming, and execution of student interviews with a team of 3 for 4 weeks in Spring 2019.

Surveys

• To determine product efficacy:

  • We used surveys to collect data from teachers and students in the control and experimental groups at 3 timepoints: before the program, immediately at its completion, and the following semester. This data helped us determine the product’s impact.

    • These surveys were the trickiest to organize and manage as they involved coordinating logistics for multiple classrooms at a time, over 3 day spans for each timepoint, for each school.

• To determine UX and room for improvement:

  • Surveys were delivered at various points to gain feedback from teachers about their experience with the program and with our training and support.

Other methods: diary entries, student feedback through their work on projects, quantitative data such as online tool usage logs and analytics were also used.

Our research highlighted these pain points and realizations

1. Newly onboarding research assistants are confused and lack confidence in how to be helpful during the complex survey administration process. (data sources: observation, self-report, interview)

2. A portion of teachers are not confident in their grasp of how to teach Brainology. (data sources: observation, self-report, interview, survey)

3. Teachers and students struggle with components in the workbooks. There are a few typos, references students don’t understand (“Who is Oprah?!”), and the organization and contents of the teacher’s implementation guide is confusing to several teachers. (data sources: observation, interview, journal entry)

So, I advocated for, spearheaded, and led the following impactful projects:

1. Creation of a Survey Administration Protocol for the internal team (2017), which improved confidence and ability of research assistants, allowing for 3 new hires to rapidly learn to run survey admin autonomously, freeing up associates’ time for back-end work.

2. Redevelopment of teacher training workshop slides (2018), after training one cohort and before training the next which led to:

   1. Higher confidence in ability to teach the material, reported by teachers in the second + later cohorts compared to first cohort.

   2. Greater observed grasp of the material by teachers and higher quality lessons taught, by teachers trained by newer workshop.

   3. Greater observed satisfaction of clients during the new training than during the original workshop.

3. A large scale redesign of the student, teacher, and teacher Guide to Implementation workbooks (2019), to fix typos; update a few culturally relevant references (“Oprah” -> “Malala”, “texting”); slightly reorganize and add minor support components for teachers; create an additional, fillable PDF version of the student workbook for online accessibility; and combine teacher answer key with teacher Guide to Implementation to eliminate need for 2 redundant books.

   • Once aligned on changes we wanted to make (team of 4 researchers, 1 director, & me - project lead, with project owner approval) the project was mostly completed by me and 1 designer on the opposite coast.

   • Timeline: approx. 3 months

   • This was completed at the end of the efficacy study and end of my contract. It aimed to improve usability, engagement, and satisfaction for new customers, and is currently available to them. Benchmark studies will be conducted by other teams.

Learnings and Takeaways

The efficacy trial concluded in early 2020. A private team is analyzing findings and will publish in 2021.

In 2019 I was commended by my Research Director, Sylvia Roberts, and CEO, Lisa Blackwell, for my initiative and impressive project and team management of translating UX findings into actionable design changes in the workbook.

Through this work, I learned the value of, as well as how to balance, collecting feedback data from clients in the mode they prefer it (in this case, they preferred to talk in person, after class) and in the most organized way for analysis (which is written or survey). I learned how to be comfortable and flexible in working with a complex research project with multiple moving parts by being organized and finding ways to align team members. I also learned how to use a toolkit of resources, in conjunction with a growth mindset of my own, to run compassionate and high fidelity research; to grow and improve through iteration; and to build and maintain positive and empathetic relationships with clients.