Applying User Research to Online K-12 Professional Development


A spike in Canvas help tickets looked like an onboarding problem, and user research showed it was really a question about what quality meant to two very different kinds of learners.

  • Project Type: User research and course framework redesign
  • Audience or Client: K-12 educators taking self-paced PD through Compass Partners in Learning’s Canvas LMS (then called TIE; a division of Black Hills Special Services Cooperative)
  • Timeline: 2023
  • Role: UX researcher and designer
  • Collaborators: Internal design team, organization leadership, educator participants in co-design sessions and interviews
  • Format or Deliverables: Two research plans, two learner personas, a two-tier course framework (Evergreen modular courses and Seasonal book studies), Figma templates, and a fillable design tool for the team

The Situation

Self-paced online courses for K-12 educators on our Canvas instance were generating an unusually high number of help tickets. When I looked closer, very few of them were about the content. Most were about getting around: where to start, how to reach the main assignment, what counted toward credit. New users struggled the most. The courses were holding up as learning experiences and falling short at the more basic job of being easy to use, and each ticket represented a teacher who had given up part of an evening to ask us something the course should have told them.

The Approach in Brief

My background pointed me toward learning experience design, but the tickets pointed somewhere else. Learning experience design asks whether a course changes what people know and do, while UX design asks whether people can use it without friction, and nearly every complaint was about the second question. So I treated the problem as user research first. When early data showed the dissatisfaction ran across the whole platform rather than only through onboarding, I widened the research and let it define what “quality” meant to our learners before proposing a fix.

What Came Out of It

The research produced a working definition of quality eLearning as beneficial, engaging, and painless, along with two learner personas who wanted noticeably different things from a course. Leadership approved a two-tier framework built around those personas, with Evergreen modular courses for durable topics and Seasonal book studies for contemporary ones. I built Figma templates for the book study framework, tested key user flows with educators, and created a fillable design tool so the team could build within the new model.

After the rollout, help tickets fell by more than half. We added exit surveys to the courses, which gave us a steady way to hear from learners instead of waiting for the next spike in tickets. Across roughly 450 responses to date, the surveys show an all-time Net Promoter Score of 46, and “I found the course materials easy to navigate,” the problem that started all of this, averages 4.74 out of 5. The results are not even across the catalog, and a few courses score well below the rest, which is part of why the surveys matter more than the averages. The personas have had light updates as those results came in.

Key Takeaway

What looked like a usability problem was partly a definition problem. Until we could say what quality meant to the people taking the courses, any fix would have been a guess, and it turned out those people did not all mean the same thing by it.


Deeper Context

The distinction between learning experience design and UX design mattered more here than I expected going in. The two fields draw on related research methods, and I was able to bring the methods background from my master’s coursework into the project, but they measure success differently. A course can be well designed as learning and still frustrate the people trying to use it, and the help tickets were evidence of exactly that gap.

Any solution also had to work inside the organization’s real constraints. Development time was limited, the solution needed to last more than a year or two, the LMS carried a cost per seat, the team had to be able to adopt the new approach without a long ramp, and whatever we built needed to fit the larger organizational structure and the other initiatives already underway. Those constraints ended up shaping the final framework as much as the research findings did.

The Journey

The first research plan focused on onboarding, because that was where I assumed the problem lived. I developed a generative toolkit for the course welcome letter, home page, and syllabus, and ran co-design sessions with three former educators standing in for our audience. Looking back, that plan rested on an assumption rather than a clear problem statement, and a narrow focus like that can easily miss what is happening more broadly.

That is roughly what happened. Partway through the first phase, new organization-wide data showed that customers felt their experience could improve across every part of the platform, not only at the start of a course. Leadership asked for a more comprehensive research plan, and the conversations that followed kept returning to the word “quality.” It became clear that we did not share a definition of it, and that different groups of users might hold different and even contradictory ones.

The second plan was built to find out. It had three parts: 1) an open questionnaire sent to every user in the Canvas instance, 2) an invitation for respondents to join video conversations about their experience, and 3) in-depth interviews with the four users who volunteered for follow-up. Out of that work came the working definition of quality as beneficial, engaging, and painless, and two personas.

The first persona, the Traveler, wanted a clear starting page, accessible and mobile-friendly courses, and a detailed home page with FAQs and support links. Travelers liked choice in assignments and page content and did not care for discussion or opportunities to share. The second persona, the Sightseer, wanted original ideas, visuals, video, and cited research. Sightseers preferred simple home pages, liked reflection questions and extra resources, and were less concerned with accessibility and mobile design. Across both, people consistently asked for clear outcomes, visuals, timely feedback, a course orientation, and consistent design, and showed less interest in discussion forums, sharing ideas, integration with their own work, and detailed rubrics. Video split along persona lines.

From there, we ran a SWOT analysis and a competitive analysis of other self-paced professional development offerings, weighing each idea against the constraints above.

The proposal leadership approved had two tiers. Evergreen courses were modular, mobile-friendly, built on topics unlikely to change quickly, and scored mostly through automated quizzes with rich feedback. They took longer to develop and were designed with Travelers in mind. Seasonal book studies were built around contemporary books, scored mostly on the completion of reflection journals, and came with optional takeaways and job aids. They were faster to build, doubled as internal PD for our own team, and were designed with Sightseers in mind.

Before rollout, I built templates of the book study framework in Figma and tested two key user flows with educators: finding the credit requirements and navigating to the main assignment. I also built a design tool as a fillable slide deck, so team members could plan courses within the two-tier model using the spiral model of instructional design, and tested its key elements with one team member. The rollout itself was a presentation to the full team with a slide deck, props representing the two user groups, and work time to start on the new process, with a clear path for moving existing courses from the old framework to the new one.

The reception of that presentation taught me something on its own. Many on the team came from an educational research background, and a questionnaire plus four interviews did not look like research by those standards. That gap in expectations was something I had not planned for.

The Work Itself

The full portfolio document for this project is available below. It includes the Traveler and Sightseer persona visuals, the diagram of the Evergreen and Seasonal tiers, and the Figma book study templates used to test the credit-requirement and main-assignment flows with educators.

What I’d Change

I would start with a clearly defined problem statement and research questions instead of a hypothesis about onboarding. The first plan was not wasted, but a sharper starting question would have pointed the research at the whole platform sooner and saved a phase.

I would also be more careful about how the personas traveled. Two personas drawn from a questionnaire and four interviews are a useful starting point, but once they were in front of the team they risked becoming settled conclusions about users that the research could not support. I should have been explicit about their limits from the first time I shared them. Reading them now, I also notice that accessibility and mobile design show up as something one persona cared about and the other cared less about. That was an honest report of what people told us, but I’m less comfortable treating accessibility as a preference that varies by learner than I was then, and I’d want the framework to make clearer that it applies to every course either way.

The team should have been involved earlier. Leadership appreciated the market thinking in the two-tier proposal, but the people who would build within it saw it fully formed for the first time at rollout, which made it harder to build a coalition around it. The one-on-one follow-ups I planned with team members after the rollout also had to be tabled.

Finally, I would change how I presented the research. Next time, I would open with a conversation about design biases, compare educational research and user research directly, and frame user research as something close to the practical action research many educators already know. I would also bring more of the evidence that started the project, meaning the help tickets and organizational data, so the findings had firmer ground under them.

Connections

For another case built on user research methods, see Exploratory Research for a State Social Studies Resource Website. For why fewer, clearer features usually serve learners better, see The Hidden Cost of EdTech Complexity, and for a way of looking at whether a course’s interactions are doing learning work, see Is All This Clicking Helping? A Calm Lens on Interactive EdTech.

If you’re working through something similar with your own online PD, you can reach me at licht.education@gmail.com, and there are more tools, articles, and resources at bradylicht.com.


Discover more from Brady Licht

Subscribe to get the latest posts sent to your email.