Week 6: Oct 17th, 2011

Usability Evaluation Framework for Caribbean Quest:

As with all software evaluation, it is impossible to maximize all 3 desirable attributes simultaneously of a UEM for educational applications.

• Generalizability
• Precision
• Realism

There are bound to be trade-offs. The goal here is the implement a comprehensive and practical (due to limited resources) evaluation method that will help generate an engaging and effective educational computer game. Some of these trade-offs can be offset using multiple methods, which is the approach we will be using in evaluating Caribbean Quest.

In their research, Omar and Jaafar identify 3 important challenges to overcome when evaluating learning/teaching software:

• Evaluation Criteria
• The Evaluators
  
• The Evaluation process

Evaluation criteria for educational computer games must go beyond functional usability. It must cover basic UI principles that help shape an appealing interface, as well as some pedagogical issues, such as providing a clear goal/objective, motivation, learners control & values. A set of criteria for educational games has been derived from Nielsen's general heuristics by Pinelle, Wong and Stach. The categories of issue are identified to be the Interface, Education/Pedagogical, Content, Playability and Multimedia (see below). Based on these issues, as well as the Child Usability Heurisitcs defined in research by Alsumait Al-Osaimi, I will generate a set of usability heuristics that will be used as a guideline by our experts, as well as a foundation for developing a formative evaluation for our end users.


As for selecting proper evaluators, using a 'hybrid' method of evaluation required a rand of end (real) users and surrogate (expert) users. For the scope of this project, as well as the scope of the CSC department, we will be focusing on gathering HCI experts and game developers as the expert users. This will allow us to evaluate the technical aspects of the game, while the content and education experts will be provided by the Psychology department. This will be discussed in more detail when we approach this phase of evaluation.



Omar and Jaafar argue that the evaluation process for an educational computers should be formative. An online evaluation questionnaire, derived from our set of Heuristics as well as a set the Playability Heuristic Evaluation for Education Games (PHEG), will be made available to our users


Challenges in the Evaluation of Educational Computer Games
By Hasiash Mohamed @ Omar and Azizah Jaafar