In recent years, there has been an increasing interest in using serious games for education. Serious games, defined as games used for purposes other than entertainment [1], have various advantages over traditional didactic learning. They provide learners a safe and authentic setting to learn and a place where learners can make mistakes without the risk of adverse patient consequences [2]. Serious games have the potential to create motivated learners who actively pursue learning on their own [3].
Using serious games to educate healthcare students is not a new idea. Serious games have been used to educate medical students on the management of acute tachyarrhythmias [4], practice of cardiopulmonary resuscitation [5], and training in emergency telemedicine [6]. The results of these studies demonstrate that serious games have the potential to be an effective learning tool. Although both medicine and pharmacy are healthcare disciplines, the results from studies done on medical serious games cannot be generalised to pharmacy serious games. Compounding extemporaneous (non-commercially available) products, dispensing medications and counselling patients are some elements of pharmacy practice [7] that are not in medical games, but pharmacy students need to learn. Pharmacy educators have been encouraged to use serious games to supplement learning for pharmacy education [2], and we are no different.
In the new Pharmacy curriculum at our institution, students have the opportunity to gain various 21st century skills and competencies [8, 9] through pharmacy professional skills training activities, such as learning and innovation skills (e.g. health communication, critical thinking and problem solving, collaboration and teamwork), digital literacy skills (e.g. information literacy, information and communication technology literacy) and life and career skills (e.g. flexibility and adaptability, social and cross-cultural interaction, responsibility and accountability) (Figure 1). However, with a large cohort size of ~200 students per batch, it becomes increasingly difficult for facilitators to train, assess and provide detailed feedback on the competency skills of each student. Convinced about the tech-savviness of the Millennial generation of students [10], the department embarked on a variety of technology-enhanced learning initiatives, with the hopes of reducing the workload and manpower costs of delivering these professional skills training lessons.
Mapping of game activities to 21st century skills. Adapted from Partnership for 21st Century Learning.
Amidst the controversies as to the types of educational technologies that could replace traditional learning methods [11, 12, 13], studies had also suggested that players’ enjoyment and interest in the game could affect their learning outcomes [14]. As such, one of the major technology-enhanced learning initiatives by the department was to develop an in-house serious game, called “Retrozfect” (or RZT), for training of 21st century skills. Our research question was two-fold – would the game enhance student learning of 21st century skills in an engaging way, and whether the game should replace or supplement the traditional curriculum in pharmacy professional skills training?
This was a cross-sectional study using a self-administered survey. Thirty-four pharmacy students were recruited using purposive sampling through snowballing technique and divided into groups of 6. All participants spent one day during their December holidays after their final exams to play the game at the Pharmacy Professional Skills Development Hub at a local university in Singapore, where their counselling labs were held. Participants were included if they were full-time undergraduate students of the pharmacy course at the university and could attend the full gaming session which encompassed one day of their school holiday break. Postgraduate students, students not from the pharmacy course, and those who helped in the design and/or development of the game scenarios in some way were excluded. Participation was voluntary and did not have any bearing on the participants’ grades. The university’s Institutional Review Board’s approval was obtained to conduct the study. There were 3 pure groups (same year of study) and 2 mixed groups (different years of study) of participants.
This multiplayer online role-playing game was set in a post-apocalyptic fantasy world infested by zombies. Players took on the role of pharmacist avatars and worked both individually and collaboratively in a team to complete the game tasks in order to find a cure for humankind (Figure 2). The game was developed in English and designed using a “cognitive authenticity” approach [15]. Players would be immersed in a fantasy game setting, but the simulations of in-game scenarios were authentic to real-life practices and the increasing complexity of the game challenged the players in terms of their cognitive loads (Figure 1). Although this game was developed in a fantasy setting, authentic learning scenarios were interspersed throughout the game, in which players would have to dispense medications and counsel the villagers (virtual patient avatars) in a “potionarium” (pharmacy equivalent). Through a “choose-your-own-adventure” format, players would select a series of possible dialogue and action choices by picking up various visual and auditory cues from the game scenarios and virtual patients. They would then need to search an in-game reference resource to find out, compound and prepare the required medicinal product using the Leap Motion – a sensor device that detected the motion of the players’ hands and fingers so as to allow them to interact with the game world [16]. The outcomes of these virtual patient encounters could be positive or negative, depending on the players’ decision-making process, thus making them take responsibility and accountability for their actions. Players would need to consider various issues from the patients’ perspectives in order to successfully complete the virtual patient encounters. Several game tasks also required players to strategize and work together in order to achieve certain in-game objectives and rewards. If a wrong selection choice or course of action was made, a feedback box would pop up explaining their mistakes and players would have to replay the game scenario with penalties to their game scores. In order to provide a more immersive experience for players, certain game scenarios also allowed the choice of using the Oculus Rift Virtual Reality (VR) headset during gameplay [17].
Summary of the in-house developed multiplayer online role-playing game.
Before gameplay, participants were asked to complete a pen-and-paper pre-quiz comprising of multiple-choice questions to evaluate their initial pharmacotherapy knowledge. After gameplay, participants were asked to complete a post-quiz comprising of the same questions. Scores from both quizzes were compared to determine whether there were any changes in marks. Participants also played a game scenario using the Oculus Rift so as to gather their feedback on its use. A debrief was conducted at the end to allow participants to clarify any questions they had about the virtual patient encounters.
Participants also completed a questionnaire comprising of 25 questions. The questionnaire obtained information about the participant’s demographics, gaming interests and frequency of playing video games in the past 6 months. Questions also obtained participants’ feedback on their gaming experiences, preferences for the game elements, and the game’s role in teaching and learning through a 5-point agreement scale (1 = strongly disagree, 2 = disagree, 3 = not decided, 4 = agree, 5 = strongly agree). In addition, participants’ perceptions on the game’s usefulness in being able to achieve its learning objectives were rated on a 5-point Likert scale (1 = not at all, 2 = slightly, 3 = moderately, 4 = very much, 5 = extremely). Part of our questionnaire also comprised of questions adopted from the Game Experience Questionnaire (GEQ), which further categorized the player’s gaming experience into 7 domains (competence, sensory and imaginative immersion, flow, tension, challenge, negative affect, and positive affect) [18]. The last section was an open-ended question to seek any other feedback about the game.
Results from the questionnaires were transcribed from hardcopy forms into the Statistical Package for the Social Sciences (SPSS) version 24. Four participants were excluded from analysis as they belonged to an incomplete group that did not have 6 players. Descriptive statistics were used to describe participants’ demographics, gaming interests and frequencies, feedback on gaming experience and role of the game in teaching and learning, and gaming preferences. Friedman two-way ANOVA and Wilcoxon signed-rank tests were used to determine if participants ranked preferences for certain gaming elements higher than other related preferences (e.g., if they ranked collaborative mode higher than competitive mode), and if they learnt more about a learning objective from the game compared to the others (e.g., if they ranked taking patient history higher than pharmacotherapy of drugs). Whenever the Friedman two-way ANOVA was used, post hoc analysis using the Wilcoxon signed-rank test was also used to determine which parameters were ranked significantly higher (at a Bonferroni corrected alpha). Wilcoxon signed-rank test was also used to determine if there was any significant difference between the pre- and post-quiz median scores. Mann-Whitney U test was used to analyze the GEQ scores from the different in-game experience domains, which ranged from 0 to 4 [19], based on the participants’ pre-university education route (i.e. traditional junior college route of the Singapore-Cambridge General Certificate of Education Advanced Level (GCE ‘A’ Level) exams versus non-GCE ‘A’ Level) and the groupings of the participants (pure versus mixed groups). For all tests, statistical significance was defined as p < 0.05.
A total of 30 students who were between 19 to 24 years old participated in this study with an even mix of males and females (15/30, 50.0% each) (Table 1). Most were Chinese (29/30, 96.7%). Majority (24/30, 80.0%) entered university through the GCE ‘A’ Level route compared to the International Baccalaureate (4/30, 13.3%) and polytechnic diploma (2/30, 6.7%). Almost half (14/30, 46.7%) were in their second year of study, while one-third (11/30, 36.7%) were in their third year of study. Most participants (24/30, 80.0%) were interested in playing video games and nearly two-thirds of them (19/30, 63.3%) had played video games in the past 6 months prior to the study.
Table 1
Participant demographics.
DEMOGRAPHICS | FREQUENCY (N = 30) (%) |
---|---|
Gender | |
Male | 15 (50%) |
Female | 15 (50%) |
Race | |
Chinese | 29 (96.7%) |
Indian | 1 (3.3) |
Pre-University Education | |
GCE ‘A’ Levela | 24 (80.0%) |
International Baccalaureate | 4 (13.3%) |
Polytechnic Diploma | 2 (6.7%) |
Current year of undergraduate study | |
Year 1 | 1 (3.3%) |
Year 2 | 14 (46.7%) |
Year 3 | 11 (36.7%) |
Year 4 | 4 (13.3%) |
GAMING INTEREST & FREQUENCY | FREQUENCY (N = 30) (%) |
General interest in playing video games | |
Not interested at all | 6 (20.0%) |
Slightly interested | 2 (6.7%) |
Moderately interested | 11 (36.7%) |
Very interested | 9 (30.0%) |
Extremely interested | 2 (6.7%) |
Frequency of playing video games in the past 6 months | |
Did not play in the last 6 months | 11 (36.7%) |
Less than 6 times in the past 6 months | 4 (13.3%) |
1–3 times every month | 3 (10.0%) |
1–3 times every week | 4 (13.3%) |
4–6 times every week | 3 (10.0%) |
Once daily | 4 (13.3%) |
More than once daily | 1 (3.3%) |
a GCE ‘A’ Level: Singapore-Cambridge General Certificate of Education Advanced Level.
Over half (17/30, 56.7%) felt that the Leap Motion device was easy to use, and most agreed that the device got easier to use over time (25/30, 83.3%) (Table 2). Majority also agreed that the Leap Motion added to their immersive gaming experience (27/30, 90.0%). In contrast, only 80.0% (24/30) of participants felt that the Oculus Rift added to their immersive experience – nearly half (14/30, 46.7%) felt uncomfortable using the Oculus Rift, with 10 participants complaining of giddiness or dizziness during gameplay. All participants liked to play the game in teams, with over three-quarters preferring to play in a group that they formed by themselves (23/30, 76.7%), compared to a pre-assigned group (7/30, 23.3%). Nearly all the participants wanted more of the game scenarios to be multiplayer (29/30, 96.7%).
Table 2
Participants’ feedback on their gameplay experiences and the role of the game in the curriculum and learning of 21st century skills.
GAMEPLAY EXPERIENCE | FREQUENCY (N = 30) (%)a | ||
---|---|---|---|
AGREE | NOT DECIDED | DISAGREE | |
Statements about the use of Leap Motion: | |||
Leap Motion is intuitive to use | 22 (73.3%) | 5 (16.7%) | 3 (10.0%) |
Leap Motion added on to the immersive experience | 27 (90.0%) | 2 (6.7%) | 1 (3.3%) |
I liked using Leap Motion in the game | 21 (70.0%) | 5 (16.7%) | 4 (13.3%) |
Leap Motion got easier to use over time | 25 (83.3%) | 4 (13.3%) | 1 (3.3%) |
Overall, Leap Motion was easy to use | 17 (56.7%) | 10 (33.3%) | 3 (10.0%) |
Statements about the use of Oculus Rift: | |||
Oculus Rift added on to the immersive experience | 24 (80.0%) | 4 (13.3%) | 1 (3.3%) |
I liked using Oculus Rift in the game | 20 (66.7%) | 8 (26.7%) | 1 (3.3%) |
I felt uncomfortable using Oculus Rift | 14 (46.7%) | 9 (30.0%) | 6 (20.0%) |
Statements about team play: | |||
I would prefer to play the game in a self-formed group | 23 (76.7%) | 5 (16.7%) | 2 (6.7%) |
I would prefer to play the game in a pre-assigned group | 7 (23.3%) | 13 (43.3%) | 10 (33.3%) |
I would enjoy playing the game regardless of whether I am in a self-formed or pre-assigned group | 18 (60.0%) | 9 (30.0%) | 3 (10.0%) |
I could communicate well with my team mates | 28 (93.3%) | 2 (6.7%) | 0 (0.0%) |
I enjoyed playing with my team mates | 30 (100.0%) | 0 (0.0%) | 0 (0.0%) |
I would like more scenarios of the game to be multiplayer | 29 (96.7%) | 1 (3.3%) | 0 (0/0%) |
Statements about avatars: | |||
The avatars appeal to me | 23 (76.7%) | 5 (16.7%) | 2 (6.7%) |
There is sufficient variety of avatars to choose from | 24 (80.0%) | 1 (3.3%) | 5 (16.7%) |
Statements about gameplay and usability: | |||
Time given to play the game is sufficient for me to complete the game tasks | 29 (96.7%) | 1 (3.3%) | 0 (0.0%) |
Objectives of the game are clear to me from the beginning of the game | 16 (53.3%) | 5 (16.7%) | 9 (30.0%) |
The tasks in the game are intuitive | 17 (56.7%) | 6 (20.0%) | 7 (23.3%) |
There are sufficient instructions for me to complete the tasks | 14 (46.7%) | 8 (26.7%) | 7 (23.3%) |
Killing zombies is a fun and entertaining task that will make me want to play the game | 24 (80.0%) | 5 (16.7%) | 1 (3.3%) |
Killing zombies hinders my learning experience | 6 (20.0%) | 2 (6.7%) | 22 (73.3%) |
Killing zombies distracts me from learning the module’s objectives | 8 (26.7%) | 3 (10.0%) | 19 (63.3%) |
I would want the full game to be played as part of the pharmacy practice curriculum | 19 (63.3%) | 7 (23.3%) | 4 (13.3%) |
I would play this game more than once | 17 (56.7%) | 7 (23.3%) | 6 (20.0%) |
Role of the game in curriculum | |||
Game is more effective than current methods of instruction in our modules to help me meet the learning objectives | 6 (20.0%) | 15 (50.0%) | 8 (26.7%) |
I would like the game to replace: | |||
Lectures | 6 (20.0%) | 6 (20.0%) | 18 (60.0%) |
face-to-face counselling sessions | 8 (26.7%) | 4 (13.3%) | 18 (60.0%) |
Extemporaneous/compounding lab sessions | 4 (13.3%) | 3 (10.0%) | 23 (76.7%) |
I would like the game to supplement: | |||
Lectures | 18 (60.0%) | 4 (13.3%) | 8 (26.7%) |
Face-to-face counselling sessions | 27 (90.0%) | 1 (3.3%) | 2 (6.7%) |
Extemporaneous/compounding lab sessions | 22 (73.3%) | 4 (13.3%) | 4 (13.3%) |
The game can be used as a formal method of assessment | 4 (13.3%) | 7 (23.3%) | 16 (53.3%) |
Learning of 21st century skills | |||
Information gathering skills required when using the Herbacopoeia helped me understand how to search for information in drug references | 19 (63.3%) | 4 (13.3%) | 6 (20.0%) |
Communication with the avatar(s) was able to help me understand how to do patient history taking and obtaining relevant health information | 24 (80.0%) | 5 (16.7%) | 0 (0.0%) |
The avatars’ response was able to help me understand the appropriateness of the medication that was recommended | 24 (80.0%) | 5 (16.7%) | 1 (3.3%) |
The game prepares me well to fill prescriptions that require me to make extemporaneous preparations e.g., suspensions, creams and ointments | 16 (53.3%) | 7 (23.3%) | 4 (13.3%) |
The game provides good exercise to calculate the amount of preparations to dispense when filling a prescription | 27 (90.0%) | 1 (3.3%) | 2 (6.7%) |
The game provides good exercise to calculate the amount of ingredients needed to make a stated amount of extemporaneous preparation | 27 (90.0%) | 1 (3.3%) | 2 (6.7%) |
The briefing I received before playing the game has enhanced my learning experience with the game | 19 (63.3%) | 6 (20.0%) | 5 (16.7%) |
The debriefing I received after playing the game has helped me understand more clearly the learning objectives of the game | 22 (73.3%) | 3 (10.0%) | 3 (10.0%) |
a Percentages may not add up to 100% due to missing responses.
In terms of game usability, only half felt that the game tasks were intuitive (17/30, 56.7%) with sufficient instructions to complete the tasks (14/30, 46.7%); and the game objectives were clear to them from the beginning of gameplay (16/30, 53.3%). A large proportion liked the zombie-killing aspects of the game and indicated that this feature did not distract them from learning the module’s objectives (19/30, 63.3%) nor hinder their overall learning experience (22/30, 73.3%). In fact, over half of them did not mind playing the game more than once (17/30, 56.7%), and two-thirds wanted to play the full game as part of their pharmacy practice curriculum (19/30, 63.3%).
In general, participants liked the three-dimensional first-person view afforded during gameplay, as well as the modern fantasy setting, authentic hero-adventure storyline and in-game system rewards (Table 3). Participants liked the collaborative nature of gameplay more than its competitive aspects (p = 0.010). In addition, participants who did not go through the traditional GCE ‘A’ level route to university had higher median scores in flow (2.50 versus 1.50, p = 0.021) and positive affect (3.25 versus 2.25, p = 0.038) than those who went through their GCE ‘A’ level (Table 4). In contrast, participants in the pure groups had higher median scores in more GEQ domains compared to mixed groups, which included competence (2.75 versus 2.00, p = 0.028), sensory and imaginative immersion (3.00 versus 2.50, p = 0.012), flow (2.50 versus 1.00, p = 0.001) and positive affect (3.00 versus 2.00, p = 0.022).
Table 3
Participants’ preferences for gaming elements and perceptions on learning 21st century skills through the game.
PREFERENCES FOR GAMING ELEMENTS | MEDIAN SCORESa (IQR) | MEAN RANK | P-VALUE |
---|---|---|---|
I liked the following in-game rewards: | |||
Item grants | 4 (4, 4) | 2.45 | 0.588 |
Feedback messages | 4 (4, 4) | 2.38 | |
Plot animations and pictures | 4 (4, 4) | 2.50 | |
Unlocking mechanisms | 4 (4, 4) | 2.67 | |
I liked the following game setting: | |||
Fantasy/Medieval/Mythic | 4 (4, 4) | N/A | 0.509c |
Modern | 4 (4, 4) | N/A | |
I liked the following game storylines: | |||
Heroic | 4 (4, 4) | 1.92 | 0.338 |
Adventurer | 4 (4, 5) | 2.12 | |
Authentic | 4 (4, 4) | 1.97 | |
I liked the 3-Dimensional (First Person) view that was in the game | 4 (3, 4) | N/A | - |
I prefer alternative views | 4 (2, 4) | N/A | - |
I would like to play in the following game modes: | |||
Competitive mode | 4 (3, 4) | 1.73 | 0.010d |
Cooperative mode | 4 (4, 5) | 2.03 | |
Collaborative mode | 4 (4, 5) | 2.23 | |
LEARNING OF 21ST CENTURY SKILLS | MEDIAN SCORESb (IQR) | MEAN RANK | P-VALUE |
How much more have you learnt from playing the game: | |||
Patient counselling skills e | 4 (3,4) | 3.95 | - |
Taking a patient history (e.g., patient age, drug allergies, medical conditions) | 3 (3,4) | 3.02 | 0.002 |
Application of drug information from a monograph | 3 (3,4) | 2.95 | 0.003 |
Pharmacotherapy of drugs (e.g., indications, contraindications, adverse drug reactions, precautions) | 3 (2,3) | 2.37 | <0.001 |
Skills in extemporaneous preparation | 3 (2.75,4) | 2.72 | 0.001 |
a Parameters were ranked on a 5-point Likert scale; 1 = Strongly disagree, 2 = Disagree, 3 = Not decided, 4 = Agree, 5 = Strongly agree.
b Parameters were ranked on a 5-point Likert scale; 1 = Not at all, 2 = Slightly, 3 = Moderately, 4 = Very much, 5 = Extremely.
c Friedman’s two-way ANOVA was conducted for all parameters except for game setting, in which Wilcoxon signed-rank test was conducted instead.
d Post hoc analysis conducted. Participants ranked “would like to play the game in a collaborative mode” higher than “would like to play the game in a competitive mode” (p = 0.010).
e Post hoc analysis conducted. Participants ranked how much more they learnt about patient counselling skills compared to the other learning objectives.
IQR: Interquartile range
N/A: Not applicable
Table 4
Analysis of GEQ domains between different participant groupings.
DOMAINS OF GEQ | MEDIAN SCORES (IQR) | P-VALUE | MEDIAN SCORES (IQR) | P-VALUE | ||
---|---|---|---|---|---|---|
GCE ‘A’ LEVEL ROUTE TO UNIVERSITY | NON- GCE ‘A’ LEVEL ROUTE TO UNIVERSITY | PURE GROUP | MIXED GROUP | |||
Competence | 2.50 (2.00, 3.00) |
2.75 (1.88, 3.50) |
0.462 | 2.75 (2.38, 3.00) |
2.00 (1.50, 2.50) |
0.028 |
Sensory and imaginative immersion | 2.50 (2.13, 3.00) |
3.00 (2.50, 3.63) |
0.210 | 3.00 (2.50, 3.50) |
2.50 (2.00, 2.88) |
0.012 |
Flow | 1.50 (1.00, 2.50) |
2.50 (2.00, 3.25) |
0.021 | 2.50 (1.50, 2.63) |
1.00 (1.00, 1.50) |
0.001 |
Tension | 1.50 (1.00, 2.38) |
1.00 (0.88, 1.25) |
0.143 | 1.50 (1.00, 2.00) |
1.50 (0.63, 2.38) |
0.692 |
Challenge | 2.25 (2.00, 3.00) |
3.00 (1.88, 3.63) |
0.251 | 2.75 (2.00, 3.13) |
2.00 (1.63, 2.75) |
0.059 |
Negative affect | 1.25 (1.00, 2.00) |
1.00 (0.75, 1.13) |
0.251 | 1.00 (1.00, 1.50) |
1.50 (0.63, 2.00) |
0.325 |
Positive affect | 2.25 (1.50, 3.00) |
3.25 (2.38, 3.63) |
0.038 | 3.00 (2.00, 3.50) |
2.00 (1.00, 2.50) |
0.022 |
GCE ‘A’ Level: General Certificate of Education Advanced Level
IQR: Interquartile range
Most participants agreed that the game was a good exercise for them to calculate the amount of ingredients for extemporaneous product preparation, and amount of preparations when dispensing a prescription (27/30, 90.0% each) (Table 2). A smaller proportion felt that the game helped them understand how to conduct patient history-taking and recommend appropriate medications (24/30, 80.0% each). Only a modest proportion felt that the game helped them in information gathering skills (19/30, 63.3%) and making extemporaneous products (16/30, 53.3%). Interestingly, participants ranked that they learnt more about patient counselling (mean rank = 3.95) than patient history-taking (mean rank = 3.02, p = 0.002), applying drug information from monographs (mean rank = 2.95, p = 0.003), pharmacotherapy of drugs (mean rank = 2.37, p < 0.001), and extemporaneous preparation skills (mean rank = 2.72, p = 0.001) (Table 3). However, there was no significant difference in median scores for the pre- and post-quiz [Median (IQR) = 6.5 (5,7) versus Median (IQR) = 6 (5,7.25); p = 0.479]. Large proportions of participants indicated that the pre-game briefing (19/30, 63.3%) and post-game debriefing (22/30, 73.3%) helped clarify the learning objectives of the game and enhance their learning experiences. Majority agreed that the game should not replace the traditional pharmacy professional skills training curriculum (≥18/30, ≥60.0%), but should supplement face-to-face counselling sessions (27/30, 90.0%), extemporaneous/compounding labs (22/30, 73.3%) and lectures (18/30, 60.0%). Only a small proportion (4/30, 13.3%) indicated that the game should be included as part of formal assessment.
Studies have shown that students need to like a serious game in order for them to be motivated learners and be interested in the game’s content [20]. Our in-house developed game (RZT) was designed with the preferences and motivations of our pharmacy students in mind. Our previous studies suggested that students liked to play a fantasy game with an adventurer storyline and an unlocking mechanism in-game reward system; viewed in a three-dimensional perspective and played in a collaborative game style [21, 22]. From these findings, we decided to develop RZT using a blended learning approach, building on De Freita’s Four-dimensional Gamification Framework and an Authentic Learning Environment [23, 24]. We deliberately created a fantasy world and storyline in RZT to pique the interest of students yet allowing them to learn the skills they needed through a “cognitive authenticity” approach, whereby they had to complete game tasks that were authentic to what they should know in actual pharmacy practices. To this end, the in-game feedback and post-game debrief played critical roles in clarifying the learning objectives for students. The results of this study reinforced our previous findings and showed that we were on the correct trajectory of game development. In fact, the name “Retrozfect” was conceived from one of the students who reflected that he had gained the knowledge and skills RETROspectively by trying to save humankind from the Zombie inFECTion. Nonetheless, we could not emphasize more on the importance of continuous involvement of end users to educators, educational designers and developers of digital education products/systems.
Among the 21st century skillsets, the most useful skillsets that students managed to learn from RZT were health communication (patient counselling), and critical thinking (pharmaceutical calculations). Interestingly, students ranked patient counselling skills higher than history-taking skills, even though a high proportion had indicated that the game helped them understand how to perform history-taking. We postulate that this was due to the provision of detailed feedback on what went wrong when students made a mistake during counselling. In comparison, the virtual patients could only provide answers to the questions asked by students during the history-taking stage, as they would need to use the information obtained for the later stages of gameplay (product preparation, dispensing and counselling).
Students ranked information literacy skills (applying drug information from monographs and learning pharmacotherapy of drugs) lower than health communication skills. Through the “cognitive authenticity” concept, herbs in the game were given hypothetical nicknames in line with the fantasy setting, but their parameters (e.g., indications, side-effects, pharmacokinetics, pharmacodynamics, storage conditions, counselling points, etc) mimicked real drugs. For example, paracetamol/acetaminophen was nicknamed “Pyrotolamol” and sildenafil was nicknamed “Kaijunhorn” instead. Our team debated long and hard over whether real drug names or hypothetical nicknames should be used in RZT, but decided to go for the latter for 2 main reasons – (i) the game was meant to be implemented in the lower year undergraduate curriculum which was mainly focused on pharmaceutical chemistry and pharmaceutics/dosage form design, thus students would not have encountered many drugs by the time they played the game; and (ii) for certain herbs in the game, we had to add in some fantasy elements so that the game storyline would not be disjointed (e.g., the indication for anti-zombie preparations was to “attack the zombie virus”, but its mechanism of action mimicked that of anti-retrovirals). Although our results showed that, in general, students felt that RZT was able to help them understand how to recommend appropriate drugs, the lower ranking of information literacy skills by students made us aware that it would be essential to discuss the drug analogies in greater detail and tease out the real and fake parameters for students during debrief when the actual game is implemented in class.
Our results did not show any significant improvement in the pre- and post-quiz scores. This could be because the quizzes were mainly assessing the students’ knowledge on the information literacy skills. The quizzes consisted of 10 multiple-choice questions, in which students had a 25% chance of guessing the correct answer. Furthermore, as this gaming session was based on voluntary recruitment of students after their exams, they might not have taken the test seriously since the scores would not impact their grades in any way. Most of them would have participated in this study to have some fun with their friends instead. This was supported by the large proportion (53.3%) who indicated that they did not want the game to be part of their formal assessments. Students felt that they learned most about health communication and critical-thinking from gameplay, therefore a consideration moving forward would be to use standardized rubrics and simulated patient-assessors in mock counselling and dispensing scenarios [25, 26], which might be more appropriate to assess students’ competency in these skillsets.
It was surprising that even though most students had indicated that RZT trained them in pharmaceutical calculations, the learning of extemporaneous preparation skills was still ranked the lowest among all the learning objectives. This mismatch could be because they did not consider the exercises on pharmaceutical calculations as part of their holistic training in extemporaneous preparation skills. Instead, they were fixated on the actual actions of making the products using the leap motion (e.g., pounding, stirring, mixing, grinding, pouring). The low ranks could have been compounded by the fact that many of them were using the Leap Motion for the first time, thus contributing to a steep learning curve. Moreover, doing hand actions in “thin air” might not have felt as realistic as holding actual compounding apparatus. Additionally, whenever they made a mistake in preparing the products, they had to repeat the whole process again, which not only impacted on their game score, but caused more frustration and dissatisfaction. If such usability issues were not addressed, it could lead to the game losing its motivational effects [27]. Our results were similar to other studies demonstrating that participants would become more familiar with using the Leap Motion with practice [28]. Hence, an in-game tutorial that familiarized players with the Leap Motion could be incorporated in future game iterations.
The GEQ scores of several domains, such as competence, sensory and imaginative immersion, flow and positive affect, were higher in pure groups than mixed groups. This was no surprise as studies have suggested that gameplay would be more enjoyable and motivating when players play with people who are familiar [29, 30]. Participants in pure groups were more likely to be more familiar with each other since they were from the same year of study. The Oculus Rift had also added on to their immersive experience. However, almost half of the participants complained of giddiness after using the device, similar to other studies [31]. In order to be inclusive of all students, 2 versions of the game could be developed – with and without the use of the device. We would also need to consider the possibility for students to form their own groups instead when the game is introduced into the curriculum.
When our game was presented at various conferences [32, 33, 34, 35], we could see that attendees were interested to adapt our game for their own teaching purposes. As our game was developed with the intention to store our students’ academic information (e.g. grades) through in-game assessments, it had to be played within the university’s secure intranet servers. Hence, other institutions would need to modify the game to fit their own institution servers. Furthermore, in this game, we had only created five scenarios that were specific to the pharmacy setting and focused on what our students needed to know in the local Singapore context. In order to cater the game for other healthcare students, there is a need to develop a wider variety of scenarios that are contextualized to the different healthcare professions in different countries. This is also an opportunity for multidisciplinary collaborations so that the game can potentially be developed and used for education among partner institutions in the future.
In light of the recent coronavirus disease (COVID-19), universities worldwide had to suspend face-to-face teaching, which led many academics to explore innovative ways to digitally engage and educate their students. Serious games, such as the one described in this study, have the potential to be an engaging educational platform to improve student learning during such times. For example, there are instances in which serious games have been developed to educate healthcare students about the coronavirus, pandemic-related topics, and safe practices and behaviors [36, 37, 38, 39, 40]. In a German study identifying the perceptions of medical undergraduates regarding their learning during COVID-19, one-third had indicated that serious games should be used during the pandemic [41]. As shown by our results, incorporating multilevel, multiplayer collaborative elements within the game can make the game more interactive and engaging, and can boost the morale and cohesiveness of the players as they learn together. Furthermore, serious games have the potential to facilitate knowledge retention over a longer period of time [36]. As we move towards the post-COVID “new normal”, we encourage educators to explore serious games as an innovative method of digital teaching to engage the new generations of students entering healthcare.
A main limitation of this study was the small sample size involved; thus our results might not be generalizable to larger student cohorts and to students from other institutions. Furthermore, due to the voluntary recruitment of students after their exams, their responses might not be representative of the entire pharmacy cohort, as there was an under-representation of year 1 and 4 undergraduates. Students from the different years of study might have different perceptions on the usefulness of the game due to their variability in terms of pharmacotherapy knowledge and exposure to pharmacy practice skills. For example, year 4 students would have had more experience and practice in patient counselling by virtue of their modules. Future studies should recruit a larger cohort of participants with equal representation across all years of study.
As this study was conducted during the holiday break after the exams, the full game had to be completed in a day in order to get sufficient numbers of participants who were interested to come. In actual implementation, students would likely play the game over a period of 5 weeks or more as part of their laboratory practical sessions. It would be possible that their in-game experience might be different if the game was played over a longer period of time instead. Future studies should consider a mixed-methods approach in order to obtain both quantitative and qualitative information about the participants’ gaming experience over a few weeks. Furthermore, our academic faculty were interested when they heard about the game, therefore it is our intention to obtain feedback from the lecturers who are interested to play the game in the future.
RZT is an in-house serious game developed to train 21st century skills in pharmacy students. We managed to merge a fantasy setting with authentic game tasks to train these skills through a “cognitive authenticity” approach. Students perceived that they managed to learn the 21st century skillsets of health communication and critical thinking through the in-game patient counselling and pharmaceutical calculation scenarios. However, majority preferred the game to be used as a supplement to the traditional methods of learning. With correct implementation, RZT has the potential to become a useful supplementary teaching tool for training health communication and critical thinking skills at our institution.
The authors would like to thank Mr. John Yin Gwee Yap and Mr. Uday S. Athreya for their contributions to this project, and Mr. Mark Wong and his team at FxMedia for working with us to develop the game. This project is supported by the National University of Singapore’s Learning Innovation Fund-Technology (grants C-148-000-038-001 and C-051-000-028-511).
This project was supported by the National University of Singapore’s Learning Innovation Fund-Technology (LIFT grants C-148-000-038-001 and C-051-000-028-511).
The authors have no competing interests to declare.
Susi T, Johannesson M, Backlund P. Serious games - An overview. Technical Report HS- IKI -TR-07-001. 2007; 1–28. http://www.diva-portal.org/smash/get/diva2:2416/FULLTEXT01.pdf, (accessed 1 Sep 2019).
Cain J, Piascik P. Are serious games a good strategy for pharmacy education? Am. J. Pharm. Educ. 2015; 79: 47. DOI: https://doi.org/10.5688/ajpe79447
Mouaheb H, Fahli A, Moussetad M, Eljamali S. The serious game: What educational benefits? Procedia Soc. Behav. Sci. 2012; 46: 5502–5508. DOI: https://doi.org/10.1016/j.sbspro.2012.06.465
Kaczmarczyk J, Davidson R, Bryden D, Haselden S, Vivekananda-Schmidt P. Learning decision making through serious games. Clin. Teach. 2016; 13: 277–282. DOI: https://doi.org/10.1111/tct.12426
Creutzfeldt J, Hedman L, Felländer-Tsai L. Cardiopulmonary resuscitation training by avatars: A qualitative study of medical students’ experiences using a multiplayer virtual world. JMIR Serious Games. 2016; 4: e22. DOI: https://doi.org/10.2196/games.6448
Nicolaidou I, Antoniades A, Constantinou R, Marangos C, Kyriacou E, Bamidis P, et al. A virtual emergency telemedicine serious game in medical training: A quantitative, professional feedback-informed evaluation study. J. Med. Internet Res. 2015; 17: e150. DOI: https://doi.org/10.2196/jmir.3667
Albanese NP, Rouse MJ, Schlaifer M. Scope of contemporary pharmacy practice: Roles, responsibilities, and functions of pharmacists and pharmacy technicians. J. Am. Pharm. Assoc. 2010; 50: e35–e69. DOI: https://doi.org/10.1331/JAPhA.2010.10510
Alismail HA, McGuire P. 21st century standards and curriculum: Current research and practice. J. Educ. Pract. 2015; 6: 150–154.
Wikipedia. 21st century skills. https://en.wikipedia.org/wiki/21st_century_skills. 2019 (accessed 10 Jun 2019).
Vogels EA. Millennials stand out for their technology use, but older generations also embrace digital life. https://www.pewresearch.org/fact-tank/2019/09/09/us-generations-technology-use/, 2019 (accessed 4 Jan 2021).
Line H. Tech should not replace traditional learning, say students. https://www.timeshighereducation.com/news/tech-should-not-replace-traditional-learning-say-students/2018250.article, 2015 (accessed 4 Jan 2020).
Heick T. What technology can and cannot replace in the classroom. https://www.teachthought.com/the-future-of-learning/what-technology-can-and-cannot-replace-in-the-classroom/, 2019 (accessed 4 Jan 2020).
Hope JK. Could educational technology replace traditional schools in the future? In: Khosrow-Pour M (Ed.), Encyclopedia of Information Science and Technology, 4th ed., Hershey, PA: IGI Global, 2018; 2421–2430.
Garris R, Ahlers R, Driskell JE. Games, motivation, and learning: A research and practice model. Simulation & Gaming. 2002; 33: 441–467. DOI: https://doi.org/10.1177/1046878102238607
Codreanu E, Sommerhoff D, Huber S, Ufer S, Seidel T. Between authenticity and cognitive demand: Finding a balance in designing a video-based simulation in the context of mathematics teacher education. Teach. Teach. Educ. 2020; 95: 103146. DOI: https://doi.org/10.1016/j.tate.2020.103146
Ultraleap. Leap Motion controller. https://www.leapmotion.com/ (accessed 4 Jan 2020).
Facebook Technologies LLC. Oculus: Play the next level of gaming. https://www.oculus.com/ (accessed 1 Sep 2019).
IJsselsteijn W, Van Den Hoogen W, Klimmt C, De Kort Y, Lindley C, Mathiak K, et al. Measuring the experience of digital game enjoyment. Proceedings of Measuring Behavior. Maastricht, Netherlands: Noldus Information Tecnology Wageningen, Netherlands, 2008; 88–89.
IJsselsteijn W, Poels K, de Kort Y. The game experience questionnaire. https://pure.tue.nl/ws/files/21666907/Game_Experience_Questionnaire_English.pdf, 2008 (accessed 1 Sep 2019).
Rieber LP. Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educ. Technol. Res. Dev. 1996; 44: 43–58. DOI: https://doi.org/10.1007/BF02300540
Chang HY, Wong LL, Yap KZ, Yap KY-L. Gaming preferences, motivations, and experiences of pharmacy students in Asia. Games Health J. 2016; 5: 40–49. DOI: https://doi.org/10.1089/g4h.2015.0028
Chang HY, Poh DY, Wong LL, Yap JY, Yap KY. Student preferences on gaming aspects for a serious game in pharmacy practice education: A cross-sectional study. JMIR Med. Educ. 2015; 1: e2. DOI: https://doi.org/10.2196/mededu.3754
Yap KY-L, Yap KZ, Yap JYG. A gamification framework for training of patient and medications management skills. J. Game Support. Interact. Learn. 2015; 1: 47–58. DOI: https://doi.org/10.15340/2148194611941
Yap KY-L, Yap KZ, Yap JYG. Creating a pharmaceutical wonderland based on gaming habits and preferences of Gen C students for the education of medication dispensing and patient counseling. Ind. Eng. Res. 2015; 8: 1–21.
Mackellar A, Ashcroft DM, Bell D, James DH, Marriott J. Identifying criteria for the assessment of pharmacy students’ communication skills with patients. Am. J. Pharm. Educ. 2007; 71: 50. DOI: https://doi.org/10.5688/aj710350
Munoz LQ, O’Byrne C, Pugsley J, Austin Z. Reliability, validity, and generalizability of an objective structured clinical examination (OSCE) for assessment of entry-to-practice in pharmacy. Pharm. Educ. 2004; 5: 1–12.
DomíNguez A, Saenz-De-Navarrete J, De-Marcos L, FernáNdez-Sanz L, PagéS C, MartíNez-HerráIz J-J. Gamifying learning experiences: Practical implications and outcomes. Computers Educ. 2013; 63: 380–392. DOI: https://doi.org/10.1016/j.compedu.2012.12.020
Han J, Gold N. Lessons learned in exploring the Leap MotionTM sensor for gesture-based instrument design. International Conference on New Interfaces for Musical Expression: Goldsmiths University of London. 2014: 371–374.
Gajadhar BJ, De Kort YA, Ijsselsteijn WA. Shared fun is doubled fun: Player enjoyment as a function of social setting. In: Markopoulos P, de Ruyter B, IJsselsteijn W, Rowland D Eds.). Fun and Games. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. 2008; 106–117. DOI: https://doi.org/10.1007/978-3-540-88322-7_11
Dolar DP. Experiments on flow and learning in games: Creating services to support efficient serious games development: Technische Universiteit Eindhoven. 2015.
Davis S, Nesbitt K, Nalivaiko E. A systematic review of cybersickness. 2014 Conference on Interactive Entertainment. 2014; 1–9. Newcastle, NSW, Australia: ACM. DOI: https://doi.org/10.1145/2677758.2677780
Yap KY-L, Yap KZ, Yap J, Athreya U. Merging fantasy and reality: Professional skills training through cognitive authenticity. S3 Conference 2019 – Asia’s Leading Healthcare Simulation Conference (2019). 22–25 Oct, Singapore. http://bit.ly/S3confRetrozfect, (accessed 17 Feb 2021).
Yap KY, Yap KZ, Yap JYG, Athreya US. Development and implementation of a 3D multiplayer online role-playing game for training of 21st century competencies and skills. Reimagine Education Conference and Awards (2017), 4–5 Dec, Philadelphia, PA, USA. http://bit.ly/2uvzZOQ, (accessed 17 Feb 2021).
Yap KY-L, Yap KZ. Training of 21st century skills through “cognitive authenticity” - Enter the Retrozfect multi-VRse. Council of Academic Public Health Institutions Australasia (CAPHIA) 2019 Australian Public Health Teaching and Learning Forum (2019), 15–17 Jul, Canberra, Australia. https://www.researchgate.net/publication/334654010_Training_of_21st_century_skills_through_cognitive_authenticity_-_Enter_the_Retrozfect_multi-VRse_CAPHIA_Australian_Teaching_and_Learning_Program_2019, (accessed 17 Feb 2021).
Yap KZ, Xie Q, Yap KY-L. How much instructions are required to play a game? An evaluation of game-based learning strategies for pharmacy professional skills. American College of Clinical Pharmacy (ACCP) Virtual Poster Symposium (2020), 26–27 May, Dallas, TX, USA. https://accp.confex.com/accp/2020vp/meetingapp.cgi/Paper/53505, (accessed 17 Feb 2021).
Hu H, Xiao Y, Li H. The effectiveness of a serious game versus online lectures for improving medical students’ coronavirus disease 2019 knowledge. Games Health J. 2021. DOI: https://doi.org/10.1089/g4h.2020.0140
Suppan M, Gartner B, Golay E, Stuby L, White M, Cottet P, et al. Teaching adequate prehospital use of personal protective equipment during the COVID-19 pandemic: Development of a gamified e-learning module. JMIR Serious Games. 2020; 8: e20173. DOI: https://doi.org/10.2196/20173
Gaspar JS, Lage EM, Silva FJD, Mineiro E, Oliveira IJR, Oliveira I, et al. A mobile serious game about the pandemic (COVID-19 - Did You Know?): Design and evaluation study. JMIR Serious Games. 2020; 8: e25226. DOI: https://doi.org/10.2196/25226
Suppan M, Catho G, Robalo Nunes T, Sauvan V, Perez M, Graf C, et al. A serious game designed to promote safe behaviors among health care workers during the COVID-19 pandemic: Development of “Escape COVID-19”. JMIR Serious Games. 2020; 8: e24986. DOI: https://doi.org/10.2196/24986
Suppan L, Abbas M, Catho G, Stuby L, Regard S, Harbarth S, et al. Impact of a serious game on the intention to change infection prevention and control practices in nursing homes during the COVID-19 pandemic: Protocol for a web-based randomized controlled trial. JMIR Res Protoc. 2020; 9: e25595. DOI: https://doi.org/10.2196/25595
Loda T, Loffler T, Erschens R, Zipfel S, Herrmann-Werner A. Medical education in times of COVID-19: German students’ expectations - A cross-sectional study. PLoS One. 2020; 15: e0241660. DOI: https://doi.org/10.1371/journal.pone.0241660