User experience evaluation framework for programming educational games

Abstract

User Experience (UX) plays an important role in educational games evaluation because it provides psychometric validation and a comprehensive gaming scale that is suitable for game evaluation purposes. Educational games have the potential to be the next educational tool in the new technology era. To assess the suitability of educational games, an established measurement framework should be used to evaluate the games. Despite readily available frameworks for UX evaluation of educational games, some still lack learning elements, while others have inadequate player content elements. Therefore, this research aimed to propose and evaluate the framework as well user experience evaluation on educational games (EDUGX), which consists of five UX elements: flow, immersion, player context, game usability, and learnability. Suitable UX elements for EDUGX were proposed based on literature review analysis, which were later reviewed by four user experience experts and a game developer before the pilot study and data collection process commenced. The assessment tool to measure students’ UX is known as the EDUGX questionnaire (EDUGXQ), consisting of 73 items. The study used an explanatory mixed methods approach as the need for quantitative (QUAN) is to obtain a wide range of responses from the selected group of students, while the need for qualitative (QUAL) is to obtain students’ detailed opinions based on the framework and support the initial data. A total of 273 diploma students in computer science participated in the data collection process, whereby they were required to play an educational computer game and then answer the measurement tool, the EDUGXQ. For the quantitative phase, exploratory factor analysis (EFA) and Rasch model analysis were applied. As for the qualitative analysis, eight students were interviewed in which the recorded interview data were later analysed using Atlas.ti. The EFA analysis showed that a total of 73 items were successfully loaded on their respective factors. Next, the Rasch analysis revealed that the EDUGXQ was an effective tool to evaluate students’ UX level with a Cronbach’s alpha value of 0.96, thus reaching the acceptable ranges of mean square outfit (MNSQ) and z-standardised outfit (ZSTD). In addition, findings from the Atlas.ti analysis revealed students’ perspectives on the importance of EDUGX elements, especially on player context, learnability and game usability. The EDUGX showed that educational and entertainment elements in educational games could be evaluated and identified. UX and game developers can use the valuable information from this research to enhance the UX evaluation for games. Therefore, EDUGX can be used by future educational game developers to ensure that all UX elements in their games are sufficient. Moreover, it is hoped that the framework and tool developed in this study can be used by UX researchers in future studies on UX evaluation.