Evaluation of "Gamified" Systems~

The paper highlights the raising trend / demand of ICT jobs but it seems that ICT job was deem as not exciting or even boring...before even getting ppl to join ICT jobs. There were efforts required to make it more interesting to find out what really happens in ICT careers. The paper sought to examine whether “gamification” could be used to broaden learners’ knowledge of CS careers and expertise.

The authors identified the components of a gamification system and the criteria that can be used for evaluating UX, usability and education usability of such a system which is attuned to the context of South Africa's education environment. Making it more interesting to learn about Computer Science related careers and hope to improve to pick up of ICT as a career choice in South Africa.

Usability evaluation 

There was good sampling of the participants (12 pax with equal in gender ratio) in terms of numbers, age and deliberate target of non-Computer Sciences students so as to target the students whom have not been exposed to IT and not familiar with the contents of the Career website. Such deliberate selection definitely reduced biases

The evaluation was conducted in the controlled setting of a computer laboratory situated in the NMMU CS Department, as a controlled setting allows the evaluator to control the users’ activity while using a system. Participants were required to complete five tasks on the CS Careers system. The participants were encouraged to ask the evaluator (who is one of the authors of this article) for assistance should they require any throughout the evaluation.

Pro there was a good sampling with gender ratio and the deliberate attempt to focus on students whom are not IT savvy

Cons participants asking the evaluator may/could generate biasness a controlled setting / environment would require resources and time to be committed to the study

Three evaluation criteria were established: Usability, UX and Education Usability

Usability

A binary scale (1 = task success, 0 = task failure) was used to measure effectiveness for some of the measured task. 

Pro a clear distinction can be seen if a task can be completed easily and further reinforced with the time taken to complete it

Cons failures wont be able to capture what are the parts the caused difficulties / failure to complete the tasks

UX and Education Usability

A Likert scale on a 5-point scale was used to measure the UX and usability results.

Pro Relatively easy to draw conclusion with the mean, standard deviation, min and max displaced for each criteria.

Cons Different individuals may have differing scale measurements.

Eventually, all three evaluation criteria - usability, UX, and education usability - received generally favorable ratings by the usability evaluation participants, suggesting that the gamification elements enhanced the CS Careers system.

References:

Brenda Scholtz, Larissa Raga, Gavin Baxter 2016, Design and Evaluation of a "Gamified" System for Improving Career Knowledge in Computer Sciences