Usability Study of Educational Robots
Rethinking Usability for Tangible
Educational Robots
Duration:
7 months
Methods:
Usability Testing, Heuristics Evaluation,
Statistical Analysis
Team Members:
Sumayya & Anjali
Tools Used:
Jasp, Qualitrics, Google Forms, Miro, Excel
Year:
2025 - 2026
Project Overview
This research project explored the usability of educational coding robots through a two-phase study. We began by evaluating different robots using established usability heuristics, then used our findings to identify limitations in existing evaluation methods for tangible robots. These insights informed the development of the first version of a new heuristic framework tailored specifically for educational robots.
13
Research sessions across two study phases.
4
Tangible educational robots usability tested.
30.5%
Task completion rate difference between highest-performing robot vs. lowest-performing robot.
Problem Statement
Educational coding robots are becoming increasingly common in K–12 STEM education, yet their interaction designs are rarely evaluated or compared systematically. Most existing research focuses on whether children learn from these robots, while little attention is given to whether the robots themselves are usable or how different interaction methods influence the user experience.
THE GAP
There is no standardized way to compare robots with different interaction styles and designers lack guidance on which interaction methods create the best experience.
WHY IT MATTERS
Poor usability shifts a child's focus from learning to overcoming interaction challenges. Reducing cognitive load through better interaction design allows learners to engage more effectively with coding activities.
Our Approach
PHASE 1
Goal: Test whether robot design affects usability
Run a comparative study across 4 robots and statistically test whether robot type and interaction model significantly affects task-completion time.
Robots Studied
To ensure meaningful comparisons, we selected four commercially available educational robots that represent distinct interaction models.
Instrument Used for Usability Evaluation
Nielsen's heuristics were picked because they're established and widely applied usability instrument in HCI. A 2016 review of 70 domain-specific heuristic studies found over 80% still used Nielsen's set as their base (Joyce & Lilley, 2016), and it remains one of the most frequently cited usability evaluation tools in the field (Gonzalez-Holland et al., 2017).
Experiment Design
Scenarios
Three structured scenarios were developed with progressive levels of interaction complexity. The tasks were intentionally sequenced to assess different aspects of user understanding and system interaction.
Methodology
Key Findings
Evaluating execution time statistically provided us a preliminary quantitative and qualitative performance data based on the time each participant took to complete the task with each robot.
Null Hypothesis (H₀): There is no significant difference in Task 1 completion time among the four robots.
Alternative Hypothesis (H₁): There is a significant difference in Task 1 completion time between at least two of the robots.
CRITICAL FINDING
Missing usability dimensions while using Nielsen’s framework
Phase 1 post-task survey was built on Nielsen's 10 usability heuristics the industry standard, designed for screens. Running it against physical robots exposed real blind spots:
PHASE 2
Goal: Designing new heuristics built for tangible educational robots
Extend Nielsen's heuristics into a framework that actually accounts for spatial, tangible, and collaborative interactions.
Synthesizing the findings from phase 1
For the development of three primary sources Nielsen's usability heuristics (Nielsen, 1994), Natural User Interface (NUI) heuristics (Maike et al., 2015), and empirical insights from the first phase of this research were reviewed collectively to identify usability and interaction factors most relevant to tangible educational robots.
Card Sorting & Theme Clustering
Similar concepts and evaluation criteria from the different sources were grouped together through thematic categorization and card sorting.
New Heuristics
Following the categorization process, each group was analyzed and refined into a distinct heuristic with an accompanying definition and evaluation criteria. This process resulted in the development of 14 heuristics collectively referred to as HER (Heuristics for Educational Robots).
The HER Framework (v1.0)
Heuristics for Educational Robots
System Awareness & Spatial Wayfinding
Robot design should always make it clear what it is doing, where it is, and what will happen next through timely, understandable feedback.
Real-World Alignment
Robot should behave in ways that match users' real-world expectations, making actions feel natural, predictable, and easy to understand.
User Control & Recovery
Users should always feel in control of the robot and be able to easily undo mistakes or recover from unexpected behavior.
Error Prevention
A robot should help users avoid mistakes before they happen through clear guidance, constraints, and intuitive interaction.
Recognition-Based Learnability
A robot should make actions easy to recognize rather than remember, helping users learn naturally through interaction.
Flexible Interaction
A robot should support multiple ways of interacting, allowing both beginners and experienced users to work efficiently.
Physical Design & Usability
A robot's physical appearance should clearly communicate how it is used while encouraging confidence and engagement.
Error Recognition
When errors occur, users should quickly recognize what went wrong and understand why.
Physical Guidance & Support
A robot should provide enough built-in guidance that users can discover and understand its features without relying on external help.
Physical Interactability
Interactive parts of a robot should be obvious, easy to distinguish, and provide clear feedback when actions are performed.
Physical Comfort & Engagement
A robot should be comfortable to use, fit naturally within its environment, and keep users engaged throughout the interaction.
Collaborative Interaction
A robot should support inclusive, coordinated, and enjoyable collaboration by making group interaction easy and natural.
Connectivity Transparency
A robot should clearly communicate its connection, power, and readiness so users always know when it is ready to interact.
Experience-Driven Aesthetics
A user's perception of a robot's appearance should improve through positive interaction, not rely solely on first impressions.
Testing the new framework with experts
The experiment focused on examining how effectively HER supports the identification of usability issues, how understandable and usable the framework is for evaluators, and whether the heuristics adequately capture the interaction characteristics unique to educational robots.
Methodology
Robots Studied
The two strongest performers from Phase 1 chosen for high usability ratings and for using genuinely different interaction models, to stress-test HER across styles.
Key Findings
Graph shows where the two robots diverged most and their Mean HER score per heuristic (7 out of 14 heuristics). The rating scale was designed from −4 (does not follow) to +4 (fully follows).
PHASE 2
Post Study
Takeaways
Post-study feedback was collected through open-ended debriefing questions asking participants to reflect on their experience using HER as an evaluation instrument.
Five themes emerged from the responses.
Reflections
This project taught me that great design starts with asking the right questions. While Nielsen's heuristics remain a strong foundation, they couldn't fully capture the unique experience of interacting with educational robots. Developing HER taught me that sometimes the biggest design challenge isn't improving the product it's improving the way we evaluate it.
Let's build something worth remembering.
Great experiences are built through collaboration.
Whether you have a project in mind, an opportunity to discuss, or simply want to connect, I'd love to hear from you!
Feel free to get in touch by
sending a message.
