Start Submission Become a Reviewer

Reading: Students’ Perceptions of an In-house Developed Multiplayer Online Roleplaying Serious Game f...

Download

A- A+
Alt. Display

Original Research

Students’ Perceptions of an In-house Developed Multiplayer Online Roleplaying Serious Game for Learning of 21st Century Skills – a Cross Sectional Study

Authors:

Kevin Yi-Lwern Yap ,

Department of Public Health, School of Psychology and Public Health, La Trobe University, Melbourne (Bundoora), Victoria 3086, AU
About Kevin Yi-Lwern

kevinyap.ehealth@gmail.com

X close

Shawn Ignatius Boon Heng Tan,

Department of Pharmacy, Changi General Hospital, 2 Simei Street 3, Singapore 529889, SG
X close

Kai Zhen Yap

Department of Pharmacy, Faculty of Science, National University of Singapore, Block S4A, 18 Science Drive 4, Singapore 117543, SG
X close

Abstract

Background: An in-house serious game was developed to train pharmacy students in 21st century skills. Players work collaboratively in a “choose-your-own-adventure” format to find a cure for humankind in a post-apocalyptic fantasy world infested by zombies. Our objectives were to determine if the game would enhance student learning of 21st century skills in an engaging way, and whether the game could replace or supplement traditional teaching methods in the pharmacy professional skills training curriculum.

Methods: A self-administered questionnaire obtained information about participants’ demographics, gaming interests, frequency, experiences and preferences; the game’s role in learning 21st century skills and its role in the curriculum. A pre- and post-quiz was conducted to test on participants’ pharmacy knowledge. Descriptive statistics, Friedman two-way ANOVA, Wilcoxon signed-rank test, and Mann-Whitney U test were used for analysis.

Results: Participants preferred the three-dimensional first-person view, modern fantasy setting, authentic hero-adventure storyline, and the collaborative nature of gameplay. Majority felt that the game trained them on performing pharmaceutical calculations (27/30, 90.0%), patient history-taking and recommending appropriate medications (24/30, 80.0% each). Participants indicated that they learnt more about patient counselling (mean rank = 3.95) compared to the other 21st century skills, such as patient history-taking (mean rank = 3.02, p = 0.002), applying drug information from monographs (mean rank = 2.95, p = 0.003), pharmacotherapy of drugs (mean rank = 2.37, p < 0.001), and extemporaneous preparation skills (mean rank = 2.72, p = 0.001). Majority agreed that the game should not replace traditional methods of teaching in the curriculum (≥60.0%), but should supplement face-to-face counselling sessions (90.0%), extemporaneous/compounding labs (73.3%) and lectures (60.0%).

Conclusion: Pharmacy students perceive that our game is able to train on 21st century skills, such as health communication (patient history taking and counselling) and performing pharmaceutical calculations. With appropriate implementation, this game has the potential to become a useful supplementary teaching tool to help enhance the learning achieved by pharmacy students.

How to Cite: Yap KY-L, Tan SIBH, Yap KZ. Students’ Perceptions of an In-house Developed Multiplayer Online Roleplaying Serious Game for Learning of 21st Century Skills – a Cross Sectional Study. International Journal of Digital Health. 2021;1(1):6. DOI: http://doi.org/10.29337/ijdh.30
154
Views
30
Downloads
  Published on 25 Mar 2021
 Accepted on 25 Feb 2021            Submitted on 05 Jan 2021

Introduction

In recent years, there has been an increasing interest in using serious games for education. Serious games, defined as games used for purposes other than entertainment [1], have various advantages over traditional didactic learning. They provide learners a safe and authentic setting to learn and a place where learners can make mistakes without the risk of adverse patient consequences [2]. Serious games have the potential to create motivated learners who actively pursue learning on their own [3].

Using serious games to educate healthcare students is not a new idea. Serious games have been used to educate medical students on the management of acute tachyarrhythmias [4], practice of cardiopulmonary resuscitation [5], and training in emergency telemedicine [6]. The results of these studies demonstrate that serious games have the potential to be an effective learning tool. Although both medicine and pharmacy are healthcare disciplines, the results from studies done on medical serious games cannot be generalised to pharmacy serious games. Compounding extemporaneous (non-commercially available) products, dispensing medications and counselling patients are some elements of pharmacy practice [7] that are not in medical games, but pharmacy students need to learn. Pharmacy educators have been encouraged to use serious games to supplement learning for pharmacy education [2], and we are no different.

In the new Pharmacy curriculum at our institution, students have the opportunity to gain various 21st century skills and competencies [8, 9] through pharmacy professional skills training activities, such as learning and innovation skills (e.g. health communication, critical thinking and problem solving, collaboration and teamwork), digital literacy skills (e.g. information literacy, information and communication technology literacy) and life and career skills (e.g. flexibility and adaptability, social and cross-cultural interaction, responsibility and accountability) (Figure 1). However, with a large cohort size of ~200 students per batch, it becomes increasingly difficult for facilitators to train, assess and provide detailed feedback on the competency skills of each student. Convinced about the tech-savviness of the Millennial generation of students [10], the department embarked on a variety of technology-enhanced learning initiatives, with the hopes of reducing the workload and manpower costs of delivering these professional skills training lessons.

Figure 1 

Mapping of game activities to 21st century skills. Adapted from Partnership for 21st Century Learning.

Amidst the controversies as to the types of educational technologies that could replace traditional learning methods [11, 12, 13], studies had also suggested that players’ enjoyment and interest in the game could affect their learning outcomes [14]. As such, one of the major technology-enhanced learning initiatives by the department was to develop an in-house serious game, called “Retrozfect” (or RZT), for training of 21st century skills. Our research question was two-fold – would the game enhance student learning of 21st century skills in an engaging way, and whether the game should replace or supplement the traditional curriculum in pharmacy professional skills training?

Methods

Study Design and Setting

This was a cross-sectional study using a self-administered survey. Thirty-four pharmacy students were recruited using purposive sampling through snowballing technique and divided into groups of 6. All participants spent one day during their December holidays after their final exams to play the game at the Pharmacy Professional Skills Development Hub at a local university in Singapore, where their counselling labs were held. Participants were included if they were full-time undergraduate students of the pharmacy course at the university and could attend the full gaming session which encompassed one day of their school holiday break. Postgraduate students, students not from the pharmacy course, and those who helped in the design and/or development of the game scenarios in some way were excluded. Participation was voluntary and did not have any bearing on the participants’ grades. The university’s Institutional Review Board’s approval was obtained to conduct the study. There were 3 pure groups (same year of study) and 2 mixed groups (different years of study) of participants.

Game Description

This multiplayer online role-playing game was set in a post-apocalyptic fantasy world infested by zombies. Players took on the role of pharmacist avatars and worked both individually and collaboratively in a team to complete the game tasks in order to find a cure for humankind (Figure 2). The game was developed in English and designed using a “cognitive authenticity” approach [15]. Players would be immersed in a fantasy game setting, but the simulations of in-game scenarios were authentic to real-life practices and the increasing complexity of the game challenged the players in terms of their cognitive loads (Figure 1). Although this game was developed in a fantasy setting, authentic learning scenarios were interspersed throughout the game, in which players would have to dispense medications and counsel the villagers (virtual patient avatars) in a “potionarium” (pharmacy equivalent). Through a “choose-your-own-adventure” format, players would select a series of possible dialogue and action choices by picking up various visual and auditory cues from the game scenarios and virtual patients. They would then need to search an in-game reference resource to find out, compound and prepare the required medicinal product using the Leap Motion – a sensor device that detected the motion of the players’ hands and fingers so as to allow them to interact with the game world [16]. The outcomes of these virtual patient encounters could be positive or negative, depending on the players’ decision-making process, thus making them take responsibility and accountability for their actions. Players would need to consider various issues from the patients’ perspectives in order to successfully complete the virtual patient encounters. Several game tasks also required players to strategize and work together in order to achieve certain in-game objectives and rewards. If a wrong selection choice or course of action was made, a feedback box would pop up explaining their mistakes and players would have to replay the game scenario with penalties to their game scores. In order to provide a more immersive experience for players, certain game scenarios also allowed the choice of using the Oculus Rift Virtual Reality (VR) headset during gameplay [17].

Figure 2 

Summary of the in-house developed multiplayer online role-playing game.

Data Collection

Before gameplay, participants were asked to complete a pen-and-paper pre-quiz comprising of multiple-choice questions to evaluate their initial pharmacotherapy knowledge. After gameplay, participants were asked to complete a post-quiz comprising of the same questions. Scores from both quizzes were compared to determine whether there were any changes in marks. Participants also played a game scenario using the Oculus Rift so as to gather their feedback on its use. A debrief was conducted at the end to allow participants to clarify any questions they had about the virtual patient encounters.

Participants also completed a questionnaire comprising of 25 questions. The questionnaire obtained information about the participant’s demographics, gaming interests and frequency of playing video games in the past 6 months. Questions also obtained participants’ feedback on their gaming experiences, preferences for the game elements, and the game’s role in teaching and learning through a 5-point agreement scale (1 = strongly disagree, 2 = disagree, 3 = not decided, 4 = agree, 5 = strongly agree). In addition, participants’ perceptions on the game’s usefulness in being able to achieve its learning objectives were rated on a 5-point Likert scale (1 = not at all, 2 = slightly, 3 = moderately, 4 = very much, 5 = extremely). Part of our questionnaire also comprised of questions adopted from the Game Experience Questionnaire (GEQ), which further categorized the player’s gaming experience into 7 domains (competence, sensory and imaginative immersion, flow, tension, challenge, negative affect, and positive affect) [18]. The last section was an open-ended question to seek any other feedback about the game.

Data Analysis

Results from the questionnaires were transcribed from hardcopy forms into the Statistical Package for the Social Sciences (SPSS) version 24. Four participants were excluded from analysis as they belonged to an incomplete group that did not have 6 players. Descriptive statistics were used to describe participants’ demographics, gaming interests and frequencies, feedback on gaming experience and role of the game in teaching and learning, and gaming preferences. Friedman two-way ANOVA and Wilcoxon signed-rank tests were used to determine if participants ranked preferences for certain gaming elements higher than other related preferences (e.g., if they ranked collaborative mode higher than competitive mode), and if they learnt more about a learning objective from the game compared to the others (e.g., if they ranked taking patient history higher than pharmacotherapy of drugs). Whenever the Friedman two-way ANOVA was used, post hoc analysis using the Wilcoxon signed-rank test was also used to determine which parameters were ranked significantly higher (at a Bonferroni corrected alpha). Wilcoxon signed-rank test was also used to determine if there was any significant difference between the pre- and post-quiz median scores. Mann-Whitney U test was used to analyze the GEQ scores from the different in-game experience domains, which ranged from 0 to 4 [19], based on the participants’ pre-university education route (i.e. traditional junior college route of the Singapore-Cambridge General Certificate of Education Advanced Level (GCE ‘A’ Level) exams versus non-GCE ‘A’ Level) and the groupings of the participants (pure versus mixed groups). For all tests, statistical significance was defined as p < 0.05.

Results

Participant Demographics

A total of 30 students who were between 19 to 24 years old participated in this study with an even mix of males and females (15/30, 50.0% each) (Table 1). Most were Chinese (29/30, 96.7%). Majority (24/30, 80.0%) entered university through the GCE ‘A’ Level route compared to the International Baccalaureate (4/30, 13.3%) and polytechnic diploma (2/30, 6.7%). Almost half (14/30, 46.7%) were in their second year of study, while one-third (11/30, 36.7%) were in their third year of study. Most participants (24/30, 80.0%) were interested in playing video games and nearly two-thirds of them (19/30, 63.3%) had played video games in the past 6 months prior to the study.

Table 1

Participant demographics.


DEMOGRAPHICS FREQUENCY (N = 30) (%)

Gender

    Male 15 (50%)

    Female 15 (50%)

Race

    Chinese 29 (96.7%)

    Indian 1 (3.3)

Pre-University Education

    GCE ‘A’ Levela 24 (80.0%)

    International Baccalaureate 4 (13.3%)

    Polytechnic Diploma 2 (6.7%)

Current year of undergraduate study

    Year 1 1 (3.3%)

    Year 2 14 (46.7%)

    Year 3 11 (36.7%)

    Year 4 4 (13.3%)

GAMING INTEREST & FREQUENCY FREQUENCY (N = 30) (%)

General interest in playing video games

    Not interested at all 6 (20.0%)

    Slightly interested 2 (6.7%)

    Moderately interested 11 (36.7%)

    Very interested 9 (30.0%)

    Extremely interested 2 (6.7%)

Frequency of playing video games in the past 6 months

    Did not play in the last 6 months 11 (36.7%)

    Less than 6 times in the past 6 months 4 (13.3%)

    1–3 times every month 3 (10.0%)

    1–3 times every week 4 (13.3%)

    4–6 times every week 3 (10.0%)

    Once daily 4 (13.3%)

    More than once daily 1 (3.3%)

a GCE ‘A’ Level: Singapore-Cambridge General Certificate of Education Advanced Level.

Participants’ Gameplay Experience and Engagement

Over half (17/30, 56.7%) felt that the Leap Motion device was easy to use, and most agreed that the device got easier to use over time (25/30, 83.3%) (Table 2). Majority also agreed that the Leap Motion added to their immersive gaming experience (27/30, 90.0%). In contrast, only 80.0% (24/30) of participants felt that the Oculus Rift added to their immersive experience – nearly half (14/30, 46.7%) felt uncomfortable using the Oculus Rift, with 10 participants complaining of giddiness or dizziness during gameplay. All participants liked to play the game in teams, with over three-quarters preferring to play in a group that they formed by themselves (23/30, 76.7%), compared to a pre-assigned group (7/30, 23.3%). Nearly all the participants wanted more of the game scenarios to be multiplayer (29/30, 96.7%).

Table 2

Participants’ feedback on their gameplay experiences and the role of the game in the curriculum and learning of 21st century skills.


GAMEPLAY EXPERIENCE FREQUENCY (N = 30) (%)a

AGREE NOT DECIDED DISAGREE

Statements about the use of Leap Motion:

Leap Motion is intuitive to use 22 (73.3%) 5 (16.7%) 3 (10.0%)

Leap Motion added on to the immersive experience 27 (90.0%) 2 (6.7%) 1 (3.3%)

I liked using Leap Motion in the game 21 (70.0%) 5 (16.7%) 4 (13.3%)

Leap Motion got easier to use over time 25 (83.3%) 4 (13.3%) 1 (3.3%)

Overall, Leap Motion was easy to use 17 (56.7%) 10 (33.3%) 3 (10.0%)

Statements about the use of Oculus Rift:

Oculus Rift added on to the immersive experience 24 (80.0%) 4 (13.3%) 1 (3.3%)

I liked using Oculus Rift in the game 20 (66.7%) 8 (26.7%) 1 (3.3%)

I felt uncomfortable using Oculus Rift 14 (46.7%) 9 (30.0%) 6 (20.0%)

Statements about team play:

I would prefer to play the game in a self-formed group 23 (76.7%) 5 (16.7%) 2 (6.7%)

I would prefer to play the game in a pre-assigned group 7 (23.3%) 13 (43.3%) 10 (33.3%)

I would enjoy playing the game regardless of whether I am in a self-formed or pre-assigned group 18 (60.0%) 9 (30.0%) 3 (10.0%)

I could communicate well with my team mates 28 (93.3%) 2 (6.7%) 0 (0.0%)

I enjoyed playing with my team mates 30 (100.0%) 0 (0.0%) 0 (0.0%)

I would like more scenarios of the game to be multiplayer 29 (96.7%) 1 (3.3%) 0 (0/0%)

Statements about avatars:

The avatars appeal to me 23 (76.7%) 5 (16.7%) 2 (6.7%)

There is sufficient variety of avatars to choose from 24 (80.0%) 1 (3.3%) 5 (16.7%)

Statements about gameplay and usability:

Time given to play the game is sufficient for me to complete the game tasks 29 (96.7%) 1 (3.3%) 0 (0.0%)

Objectives of the game are clear to me from the beginning of the game 16 (53.3%) 5 (16.7%) 9 (30.0%)

The tasks in the game are intuitive 17 (56.7%) 6 (20.0%) 7 (23.3%)

There are sufficient instructions for me to complete the tasks 14 (46.7%) 8 (26.7%) 7 (23.3%)

Killing zombies is a fun and entertaining task that will make me want to play the game 24 (80.0%) 5 (16.7%) 1 (3.3%)

Killing zombies hinders my learning experience 6 (20.0%) 2 (6.7%) 22 (73.3%)

Killing zombies distracts me from learning the module’s objectives 8 (26.7%) 3 (10.0%) 19 (63.3%)

I would want the full game to be played as part of the pharmacy practice curriculum 19 (63.3%) 7 (23.3%) 4 (13.3%)

I would play this game more than once 17 (56.7%) 7 (23.3%) 6 (20.0%)

Role of the game in curriculum

Game is more effective than current methods of instruction in our modules to help me meet the learning objectives 6 (20.0%) 15 (50.0%) 8 (26.7%)

I would like the game to replace:

Lectures 6 (20.0%) 6 (20.0%) 18 (60.0%)

face-to-face counselling sessions 8 (26.7%) 4 (13.3%) 18 (60.0%)

Extemporaneous/compounding lab sessions 4 (13.3%) 3 (10.0%) 23 (76.7%)

I would like the game to supplement:

Lectures 18 (60.0%) 4 (13.3%) 8 (26.7%)

Face-to-face counselling sessions 27 (90.0%) 1 (3.3%) 2 (6.7%)

Extemporaneous/compounding lab sessions 22 (73.3%) 4 (13.3%) 4 (13.3%)

The game can be used as a formal method of assessment 4 (13.3%) 7 (23.3%) 16 (53.3%)

Learning of 21st century skills

Information gathering skills required when using the Herbacopoeia helped me understand how to search for information in drug references 19 (63.3%) 4 (13.3%) 6 (20.0%)

Communication with the avatar(s) was able to help me understand how to do patient history taking and obtaining relevant health information 24 (80.0%) 5 (16.7%) 0 (0.0%)

The avatars’ response was able to help me understand the appropriateness of the medication that was recommended 24 (80.0%) 5 (16.7%) 1 (3.3%)

The game prepares me well to fill prescriptions that require me to make extemporaneous preparations e.g., suspensions, creams and ointments 16 (53.3%) 7 (23.3%) 4 (13.3%)

The game provides good exercise to calculate the amount of preparations to dispense when filling a prescription 27 (90.0%) 1 (3.3%) 2 (6.7%)

The game provides good exercise to calculate the amount of ingredients needed to make a stated amount of extemporaneous preparation 27 (90.0%) 1 (3.3%) 2 (6.7%)

The briefing I received before playing the game has enhanced my learning experience with the game 19 (63.3%) 6 (20.0%) 5 (16.7%)

The debriefing I received after playing the game has helped me understand more clearly the learning objectives of the game 22 (73.3%) 3 (10.0%) 3 (10.0%)

a Percentages may not add up to 100% due to missing responses.

In terms of game usability, only half felt that the game tasks were intuitive (17/30, 56.7%) with sufficient instructions to complete the tasks (14/30, 46.7%); and the game objectives were clear to them from the beginning of gameplay (16/30, 53.3%). A large proportion liked the zombie-killing aspects of the game and indicated that this feature did not distract them from learning the module’s objectives (19/30, 63.3%) nor hinder their overall learning experience (22/30, 73.3%). In fact, over half of them did not mind playing the game more than once (17/30, 56.7%), and two-thirds wanted to play the full game as part of their pharmacy practice curriculum (19/30, 63.3%).

In general, participants liked the three-dimensional first-person view afforded during gameplay, as well as the modern fantasy setting, authentic hero-adventure storyline and in-game system rewards (Table 3). Participants liked the collaborative nature of gameplay more than its competitive aspects (p = 0.010). In addition, participants who did not go through the traditional GCE ‘A’ level route to university had higher median scores in flow (2.50 versus 1.50, p = 0.021) and positive affect (3.25 versus 2.25, p = 0.038) than those who went through their GCE ‘A’ level (Table 4). In contrast, participants in the pure groups had higher median scores in more GEQ domains compared to mixed groups, which included competence (2.75 versus 2.00, p = 0.028), sensory and imaginative immersion (3.00 versus 2.50, p = 0.012), flow (2.50 versus 1.00, p = 0.001) and positive affect (3.00 versus 2.00, p = 0.022).

Table 3

Participants’ preferences for gaming elements and perceptions on learning 21st century skills through the game.


PREFERENCES FOR GAMING ELEMENTS MEDIAN SCORESa (IQR) MEAN RANK P-VALUE

I liked the following in-game rewards:

    Item grants 4 (4, 4) 2.45 0.588

    Feedback messages 4 (4, 4) 2.38

    Plot animations and pictures 4 (4, 4) 2.50

    Unlocking mechanisms 4 (4, 4) 2.67

I liked the following game setting:

    Fantasy/Medieval/Mythic 4 (4, 4) N/A 0.509c

    Modern 4 (4, 4) N/A

I liked the following game storylines:

    Heroic 4 (4, 4) 1.92 0.338

    Adventurer 4 (4, 5) 2.12

    Authentic 4 (4, 4) 1.97

I liked the 3-Dimensional (First Person) view that was in the game 4 (3, 4) N/A -

I prefer alternative views 4 (2, 4) N/A -

I would like to play in the following game modes:

    Competitive mode 4 (3, 4) 1.73 0.010d

    Cooperative mode 4 (4, 5) 2.03

    Collaborative mode 4 (4, 5) 2.23

LEARNING OF 21ST CENTURY SKILLS MEDIAN SCORESb (IQR) MEAN RANK P-VALUE

How much more have you learnt from playing the game:

Patient counselling skills e 4 (3,4) 3.95 -

Taking a patient history (e.g., patient age, drug allergies, medical conditions) 3 (3,4) 3.02 0.002

Application of drug information from a monograph 3 (3,4) 2.95 0.003

Pharmacotherapy of drugs (e.g., indications, contraindications, adverse drug reactions, precautions) 3 (2,3) 2.37 <0.001

Skills in extemporaneous preparation 3 (2.75,4) 2.72 0.001

a Parameters were ranked on a 5-point Likert scale; 1 = Strongly disagree, 2 = Disagree, 3 = Not decided, 4 = Agree, 5 = Strongly agree.

b Parameters were ranked on a 5-point Likert scale; 1 = Not at all, 2 = Slightly, 3 = Moderately, 4 = Very much, 5 = Extremely.

c Friedman’s two-way ANOVA was conducted for all parameters except for game setting, in which Wilcoxon signed-rank test was conducted instead.

d Post hoc analysis conducted. Participants ranked “would like to play the game in a collaborative mode” higher than “would like to play the game in a competitive mode” (p = 0.010).

e Post hoc analysis conducted. Participants ranked how much more they learnt about patient counselling skills compared to the other learning objectives.

IQR: Interquartile range

N/A: Not applicable

Table 4

Analysis of GEQ domains between different participant groupings.


DOMAINS OF GEQ MEDIAN SCORES (IQR) P-VALUE MEDIAN SCORES (IQR) P-VALUE


GCE ‘A’ LEVEL ROUTE TO UNIVERSITY NON- GCE ‘A’ LEVEL ROUTE TO UNIVERSITY PURE GROUP MIXED GROUP

Competence 2.50
(2.00, 3.00)
2.75
(1.88, 3.50)
0.462 2.75
(2.38, 3.00)
2.00
(1.50, 2.50)
0.028

Sensory and imaginative immersion 2.50
(2.13, 3.00)
3.00
(2.50, 3.63)
0.210 3.00
(2.50, 3.50)
2.50
(2.00, 2.88)
0.012

Flow 1.50
(1.00, 2.50)
2.50
(2.00, 3.25)
0.021 2.50
(1.50, 2.63)
1.00
(1.00, 1.50)
0.001

Tension 1.50
(1.00, 2.38)
1.00
(0.88, 1.25)
0.143 1.50
(1.00, 2.00)
1.50
(0.63, 2.38)
0.692

Challenge 2.25
(2.00, 3.00)
3.00
(1.88, 3.63)
0.251 2.75
(2.00, 3.13)
2.00
(1.63, 2.75)
0.059

Negative affect 1.25
(1.00, 2.00)
1.00
(0.75, 1.13)
0.251 1.00
(1.00, 1.50)
1.50
(0.63, 2.00)
0.325

Positive affect 2.25
(1.50, 3.00)
3.25
(2.38, 3.63)
0.038 3.00
(2.00, 3.50)
2.00
(1.00, 2.50)
0.022

GCE ‘A’ Level: General Certificate of Education Advanced Level

IQR: Interquartile range

Role of Game in Learning 21st Century Skills and in the Curriculum

Most participants agreed that the game was a good exercise for them to calculate the amount of ingredients for extemporaneous product preparation, and amount of preparations when dispensing a prescription (27/30, 90.0% each) (Table 2). A smaller proportion felt that the game helped them understand how to conduct patient history-taking and recommend appropriate medications (24/30, 80.0% each). Only a modest proportion felt that the game helped them in information gathering skills (19/30, 63.3%) and making extemporaneous products (16/30, 53.3%). Interestingly, participants ranked that they learnt more about patient counselling (mean rank = 3.95) than patient history-taking (mean rank = 3.02, p = 0.002), applying drug information from monographs (mean rank = 2.95, p = 0.003), pharmacotherapy of drugs (mean rank = 2.37, p < 0.001), and extemporaneous preparation skills (mean rank = 2.72, p = 0.001) (Table 3). However, there was no significant difference in median scores for the pre- and post-quiz [Median (IQR) = 6.5 (5,7) versus Median (IQR) = 6 (5,7.25); p = 0.479]. Large proportions of participants indicated that the pre-game briefing (19/30, 63.3%) and post-game debriefing (22/30, 73.3%) helped clarify the learning objectives of the game and enhance their learning experiences. Majority agreed that the game should not replace the traditional pharmacy professional skills training curriculum (≥18/30, ≥60.0%), but should supplement face-to-face counselling sessions (27/30, 90.0%), extemporaneous/compounding labs (22/30, 73.3%) and lectures (18/30, 60.0%). Only a small proportion (4/30, 13.3%) indicated that the game should be included as part of formal assessment.

Discussion

Studies have shown that students need to like a serious game in order for them to be motivated learners and be interested in the game’s content [20]. Our in-house developed game (RZT) was designed with the preferences and motivations of our pharmacy students in mind. Our previous studies suggested that students liked to play a fantasy game with an adventurer storyline and an unlocking mechanism in-game reward system; viewed in a three-dimensional perspective and played in a collaborative game style [21, 22]. From these findings, we decided to develop RZT using a blended learning approach, building on De Freita’s Four-dimensional Gamification Framework and an Authentic Learning Environment [23, 24]. We deliberately created a fantasy world and storyline in RZT to pique the interest of students yet allowing them to learn the skills they needed through a “cognitive authenticity” approach, whereby they had to complete game tasks that were authentic to what they should know in actual pharmacy practices. To this end, the in-game feedback and post-game debrief played critical roles in clarifying the learning objectives for students. The results of this study reinforced our previous findings and showed that we were on the correct trajectory of game development. In fact, the name “Retrozfect” was conceived from one of the students who reflected that he had gained the knowledge and skills RETROspectively by trying to save humankind from the Zombie inFECTion. Nonetheless, we could not emphasize more on the importance of continuous involvement of end users to educators, educational designers and developers of digital education products/systems.

Among the 21st century skillsets, the most useful skillsets that students managed to learn from RZT were health communication (patient counselling), and critical thinking (pharmaceutical calculations). Interestingly, students ranked patient counselling skills higher than history-taking skills, even though a high proportion had indicated that the game helped them understand how to perform history-taking. We postulate that this was due to the provision of detailed feedback on what went wrong when students made a mistake during counselling. In comparison, the virtual patients could only provide answers to the questions asked by students during the history-taking stage, as they would need to use the information obtained for the later stages of gameplay (product preparation, dispensing and counselling).

Students ranked information literacy skills (applying drug information from monographs and learning pharmacotherapy of drugs) lower than health communication skills. Through the “cognitive authenticity” concept, herbs in the game were given hypothetical nicknames in line with the fantasy setting, but their parameters (e.g., indications, side-effects, pharmacokinetics, pharmacodynamics, storage conditions, counselling points, etc) mimicked real drugs. For example, paracetamol/acetaminophen was nicknamed “Pyrotolamol” and sildenafil was nicknamed “Kaijunhorn” instead. Our team debated long and hard over whether real drug names or hypothetical nicknames should be used in RZT, but decided to go for the latter for 2 main reasons – (i) the game was meant to be implemented in the lower year undergraduate curriculum which was mainly focused on pharmaceutical chemistry and pharmaceutics/dosage form design, thus students would not have encountered many drugs by the time they played the game; and (ii) for certain herbs in the game, we had to add in some fantasy elements so that the game storyline would not be disjointed (e.g., the indication for anti-zombie preparations was to “attack the zombie virus”, but its mechanism of action mimicked that of anti-retrovirals). Although our results showed that, in general, students felt that RZT was able to help them understand how to recommend appropriate drugs, the lower ranking of information literacy skills by students made us aware that it would be essential to discuss the drug analogies in greater detail and tease out the real and fake parameters for students during debrief when the actual game is implemented in class.

Our results did not show any significant improvement in the pre- and post-quiz scores. This could be because the quizzes were mainly assessing the students’ knowledge on the information literacy skills. The quizzes consisted of 10 multiple-choice questions, in which students had a 25% chance of guessing the correct answer. Furthermore, as this gaming session was based on voluntary recruitment of students after their exams, they might not have taken the test seriously since the scores would not impact their grades in any way. Most of them would have participated in this study to have some fun with their friends instead. This was supported by the large proportion (53.3%) who indicated that they did not want the game to be part of their formal assessments. Students felt that they learned most about health communication and critical-thinking from gameplay, therefore a consideration moving forward would be to use standardized rubrics and simulated patient-assessors in mock counselling and dispensing scenarios [25, 26], which might be more appropriate to assess students’ competency in these skillsets.

It was surprising that even though most students had indicated that RZT trained them in pharmaceutical calculations, the learning of extemporaneous preparation skills was still ranked the lowest among all the learning objectives. This mismatch could be because they did not consider the exercises on pharmaceutical calculations as part of their holistic training in extemporaneous preparation skills. Instead, they were fixated on the actual actions of making the products using the leap motion (e.g., pounding, stirring, mixing, grinding, pouring). The low ranks could have been compounded by the fact that many of them were using the Leap Motion for the first time, thus contributing to a steep learning curve. Moreover, doing hand actions in “thin air” might not have felt as realistic as holding actual compounding apparatus. Additionally, whenever they made a mistake in preparing the products, they had to repeat the whole process again, which not only impacted on their game score, but caused more frustration and dissatisfaction. If such usability issues were not addressed, it could lead to the game losing its motivational effects [27]. Our results were similar to other studies demonstrating that participants would become more familiar with using the Leap Motion with practice [28]. Hence, an in-game tutorial that familiarized players with the Leap Motion could be incorporated in future game iterations.

The GEQ scores of several domains, such as competence, sensory and imaginative immersion, flow and positive affect, were higher in pure groups than mixed groups. This was no surprise as studies have suggested that gameplay would be more enjoyable and motivating when players play with people who are familiar [29, 30]. Participants in pure groups were more likely to be more familiar with each other since they were from the same year of study. The Oculus Rift had also added on to their immersive experience. However, almost half of the participants complained of giddiness after using the device, similar to other studies [31]. In order to be inclusive of all students, 2 versions of the game could be developed – with and without the use of the device. We would also need to consider the possibility for students to form their own groups instead when the game is introduced into the curriculum.

When our game was presented at various conferences [32, 33, 34, 35], we could see that attendees were interested to adapt our game for their own teaching purposes. As our game was developed with the intention to store our students’ academic information (e.g. grades) through in-game assessments, it had to be played within the university’s secure intranet servers. Hence, other institutions would need to modify the game to fit their own institution servers. Furthermore, in this game, we had only created five scenarios that were specific to the pharmacy setting and focused on what our students needed to know in the local Singapore context. In order to cater the game for other healthcare students, there is a need to develop a wider variety of scenarios that are contextualized to the different healthcare professions in different countries. This is also an opportunity for multidisciplinary collaborations so that the game can potentially be developed and used for education among partner institutions in the future.

In light of the recent coronavirus disease (COVID-19), universities worldwide had to suspend face-to-face teaching, which led many academics to explore innovative ways to digitally engage and educate their students. Serious games, such as the one described in this study, have the potential to be an engaging educational platform to improve student learning during such times. For example, there are instances in which serious games have been developed to educate healthcare students about the coronavirus, pandemic-related topics, and safe practices and behaviors [36, 37, 38, 39, 40]. In a German study identifying the perceptions of medical undergraduates regarding their learning during COVID-19, one-third had indicated that serious games should be used during the pandemic [41]. As shown by our results, incorporating multilevel, multiplayer collaborative elements within the game can make the game more interactive and engaging, and can boost the morale and cohesiveness of the players as they learn together. Furthermore, serious games have the potential to facilitate knowledge retention over a longer period of time [36]. As we move towards the post-COVID “new normal”, we encourage educators to explore serious games as an innovative method of digital teaching to engage the new generations of students entering healthcare.

Limitations

A main limitation of this study was the small sample size involved; thus our results might not be generalizable to larger student cohorts and to students from other institutions. Furthermore, due to the voluntary recruitment of students after their exams, their responses might not be representative of the entire pharmacy cohort, as there was an under-representation of year 1 and 4 undergraduates. Students from the different years of study might have different perceptions on the usefulness of the game due to their variability in terms of pharmacotherapy knowledge and exposure to pharmacy practice skills. For example, year 4 students would have had more experience and practice in patient counselling by virtue of their modules. Future studies should recruit a larger cohort of participants with equal representation across all years of study.

As this study was conducted during the holiday break after the exams, the full game had to be completed in a day in order to get sufficient numbers of participants who were interested to come. In actual implementation, students would likely play the game over a period of 5 weeks or more as part of their laboratory practical sessions. It would be possible that their in-game experience might be different if the game was played over a longer period of time instead. Future studies should consider a mixed-methods approach in order to obtain both quantitative and qualitative information about the participants’ gaming experience over a few weeks. Furthermore, our academic faculty were interested when they heard about the game, therefore it is our intention to obtain feedback from the lecturers who are interested to play the game in the future.

Conclusion

RZT is an in-house serious game developed to train 21st century skills in pharmacy students. We managed to merge a fantasy setting with authentic game tasks to train these skills through a “cognitive authenticity” approach. Students perceived that they managed to learn the 21st century skillsets of health communication and critical thinking through the in-game patient counselling and pharmaceutical calculation scenarios. However, majority preferred the game to be used as a supplement to the traditional methods of learning. With correct implementation, RZT has the potential to become a useful supplementary teaching tool for training health communication and critical thinking skills at our institution.

Acknowledgements

The authors would like to thank Mr. John Yin Gwee Yap and Mr. Uday S. Athreya for their contributions to this project, and Mr. Mark Wong and his team at FxMedia for working with us to develop the game. This project is supported by the National University of Singapore’s Learning Innovation Fund-Technology (grants C-148-000-038-001 and C-051-000-028-511).

Funding Information

This project was supported by the National University of Singapore’s Learning Innovation Fund-Technology (LIFT grants C-148-000-038-001 and C-051-000-028-511).

Competing Interests

The authors have no competing interests to declare.

References

  1. Susi T, Johannesson M, Backlund P. Serious games - An overview. Technical Report HS- IKI -TR-07-001. 2007; 1–28. http://www.diva-portal.org/smash/get/diva2:2416/FULLTEXT01.pdf, (accessed 1 Sep 2019). 

  2. Cain J, Piascik P. Are serious games a good strategy for pharmacy education? Am. J. Pharm. Educ. 2015; 79: 47. DOI: https://doi.org/10.5688/ajpe79447 

  3. Mouaheb H, Fahli A, Moussetad M, Eljamali S. The serious game: What educational benefits? Procedia Soc. Behav. Sci. 2012; 46: 5502–5508. DOI: https://doi.org/10.1016/j.sbspro.2012.06.465 

  4. Kaczmarczyk J, Davidson R, Bryden D, Haselden S, Vivekananda-Schmidt P. Learning decision making through serious games. Clin. Teach. 2016; 13: 277–282. DOI: https://doi.org/10.1111/tct.12426 

  5. Creutzfeldt J, Hedman L, Felländer-Tsai L. Cardiopulmonary resuscitation training by avatars: A qualitative study of medical students’ experiences using a multiplayer virtual world. JMIR Serious Games. 2016; 4: e22. DOI: https://doi.org/10.2196/games.6448 

  6. Nicolaidou I, Antoniades A, Constantinou R, Marangos C, Kyriacou E, Bamidis P, et al. A virtual emergency telemedicine serious game in medical training: A quantitative, professional feedback-informed evaluation study. J. Med. Internet Res. 2015; 17: e150. DOI: https://doi.org/10.2196/jmir.3667 

  7. Albanese NP, Rouse MJ, Schlaifer M. Scope of contemporary pharmacy practice: Roles, responsibilities, and functions of pharmacists and pharmacy technicians. J. Am. Pharm. Assoc. 2010; 50: e35–e69. DOI: https://doi.org/10.1331/JAPhA.2010.10510 

  8. Alismail HA, McGuire P. 21st century standards and curriculum: Current research and practice. J. Educ. Pract. 2015; 6: 150–154. 

  9. Wikipedia. 21st century skills. https://en.wikipedia.org/wiki/21st_century_skills. 2019 (accessed 10 Jun 2019). 

  10. Vogels EA. Millennials stand out for their technology use, but older generations also embrace digital life. https://www.pewresearch.org/fact-tank/2019/09/09/us-generations-technology-use/, 2019 (accessed 4 Jan 2021). 

  11. Line H. Tech should not replace traditional learning, say students. https://www.timeshighereducation.com/news/tech-should-not-replace-traditional-learning-say-students/2018250.article, 2015 (accessed 4 Jan 2020). 

  12. Heick T. What technology can and cannot replace in the classroom. https://www.teachthought.com/the-future-of-learning/what-technology-can-and-cannot-replace-in-the-classroom/, 2019 (accessed 4 Jan 2020). 

  13. Hope JK. Could educational technology replace traditional schools in the future? In: Khosrow-Pour M (Ed.), Encyclopedia of Information Science and Technology, 4th ed., Hershey, PA: IGI Global, 2018; 2421–2430. 

  14. Garris R, Ahlers R, Driskell JE. Games, motivation, and learning: A research and practice model. Simulation & Gaming. 2002; 33: 441–467. DOI: https://doi.org/10.1177/1046878102238607 

  15. Codreanu E, Sommerhoff D, Huber S, Ufer S, Seidel T. Between authenticity and cognitive demand: Finding a balance in designing a video-based simulation in the context of mathematics teacher education. Teach. Teach. Educ. 2020; 95: 103146. DOI: https://doi.org/10.1016/j.tate.2020.103146 

  16. Ultraleap. Leap Motion controller. https://www.leapmotion.com/ (accessed 4 Jan 2020). 

  17. Facebook Technologies LLC. Oculus: Play the next level of gaming. https://www.oculus.com/ (accessed 1 Sep 2019). 

  18. IJsselsteijn W, Van Den Hoogen W, Klimmt C, De Kort Y, Lindley C, Mathiak K, et al. Measuring the experience of digital game enjoyment. Proceedings of Measuring Behavior. Maastricht, Netherlands: Noldus Information Tecnology Wageningen, Netherlands, 2008; 88–89. 

  19. IJsselsteijn W, Poels K, de Kort Y. The game experience questionnaire. https://pure.tue.nl/ws/files/21666907/Game_Experience_Questionnaire_English.pdf, 2008 (accessed 1 Sep 2019). 

  20. Rieber LP. Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educ. Technol. Res. Dev. 1996; 44: 43–58. DOI: https://doi.org/10.1007/BF02300540 

  21. Chang HY, Wong LL, Yap KZ, Yap KY-L. Gaming preferences, motivations, and experiences of pharmacy students in Asia. Games Health J. 2016; 5: 40–49. DOI: https://doi.org/10.1089/g4h.2015.0028 

  22. Chang HY, Poh DY, Wong LL, Yap JY, Yap KY. Student preferences on gaming aspects for a serious game in pharmacy practice education: A cross-sectional study. JMIR Med. Educ. 2015; 1: e2. DOI: https://doi.org/10.2196/mededu.3754 

  23. Yap KY-L, Yap KZ, Yap JYG. A gamification framework for training of patient and medications management skills. J. Game Support. Interact. Learn. 2015; 1: 47–58. DOI: https://doi.org/10.15340/2148194611941 

  24. Yap KY-L, Yap KZ, Yap JYG. Creating a pharmaceutical wonderland based on gaming habits and preferences of Gen C students for the education of medication dispensing and patient counseling. Ind. Eng. Res. 2015; 8: 1–21. 

  25. Mackellar A, Ashcroft DM, Bell D, James DH, Marriott J. Identifying criteria for the assessment of pharmacy students’ communication skills with patients. Am. J. Pharm. Educ. 2007; 71: 50. DOI: https://doi.org/10.5688/aj710350 

  26. Munoz LQ, O’Byrne C, Pugsley J, Austin Z. Reliability, validity, and generalizability of an objective structured clinical examination (OSCE) for assessment of entry-to-practice in pharmacy. Pharm. Educ. 2004; 5: 1–12. 

  27. DomíNguez A, Saenz-De-Navarrete J, De-Marcos L, FernáNdez-Sanz L, PagéS C, MartíNez-HerráIz J-J. Gamifying learning experiences: Practical implications and outcomes. Computers Educ. 2013; 63: 380–392. DOI: https://doi.org/10.1016/j.compedu.2012.12.020 

  28. Han J, Gold N. Lessons learned in exploring the Leap MotionTM sensor for gesture-based instrument design. International Conference on New Interfaces for Musical Expression: Goldsmiths University of London. 2014: 371–374. 

  29. Gajadhar BJ, De Kort YA, Ijsselsteijn WA. Shared fun is doubled fun: Player enjoyment as a function of social setting. In: Markopoulos P, de Ruyter B, IJsselsteijn W, Rowland D Eds.). Fun and Games. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. 2008; 106–117. DOI: https://doi.org/10.1007/978-3-540-88322-7_11 

  30. Dolar DP. Experiments on flow and learning in games: Creating services to support efficient serious games development: Technische Universiteit Eindhoven. 2015. 

  31. Davis S, Nesbitt K, Nalivaiko E. A systematic review of cybersickness. 2014 Conference on Interactive Entertainment. 2014; 1–9. Newcastle, NSW, Australia: ACM. DOI: https://doi.org/10.1145/2677758.2677780 

  32. Yap KY-L, Yap KZ, Yap J, Athreya U. Merging fantasy and reality: Professional skills training through cognitive authenticity. S3 Conference 2019 – Asia’s Leading Healthcare Simulation Conference (2019). 22–25 Oct, Singapore. http://bit.ly/S3confRetrozfect, (accessed 17 Feb 2021). 

  33. Yap KY, Yap KZ, Yap JYG, Athreya US. Development and implementation of a 3D multiplayer online role-playing game for training of 21st century competencies and skills. Reimagine Education Conference and Awards (2017), 4–5 Dec, Philadelphia, PA, USA. http://bit.ly/2uvzZOQ, (accessed 17 Feb 2021). 

  34. Yap KY-L, Yap KZ. Training of 21st century skills through “cognitive authenticity” - Enter the Retrozfect multi-VRse. Council of Academic Public Health Institutions Australasia (CAPHIA) 2019 Australian Public Health Teaching and Learning Forum (2019), 15–17 Jul, Canberra, Australia. https://www.researchgate.net/publication/334654010_Training_of_21st_century_skills_through_cognitive_authenticity_-_Enter_the_Retrozfect_multi-VRse_CAPHIA_Australian_Teaching_and_Learning_Program_2019, (accessed 17 Feb 2021). 

  35. Yap KZ, Xie Q, Yap KY-L. How much instructions are required to play a game? An evaluation of game-based learning strategies for pharmacy professional skills. American College of Clinical Pharmacy (ACCP) Virtual Poster Symposium (2020), 26–27 May, Dallas, TX, USA. https://accp.confex.com/accp/2020vp/meetingapp.cgi/Paper/53505, (accessed 17 Feb 2021). 

  36. Hu H, Xiao Y, Li H. The effectiveness of a serious game versus online lectures for improving medical students’ coronavirus disease 2019 knowledge. Games Health J. 2021. DOI: https://doi.org/10.1089/g4h.2020.0140 

  37. Suppan M, Gartner B, Golay E, Stuby L, White M, Cottet P, et al. Teaching adequate prehospital use of personal protective equipment during the COVID-19 pandemic: Development of a gamified e-learning module. JMIR Serious Games. 2020; 8: e20173. DOI: https://doi.org/10.2196/20173 

  38. Gaspar JS, Lage EM, Silva FJD, Mineiro E, Oliveira IJR, Oliveira I, et al. A mobile serious game about the pandemic (COVID-19 - Did You Know?): Design and evaluation study. JMIR Serious Games. 2020; 8: e25226. DOI: https://doi.org/10.2196/25226 

  39. Suppan M, Catho G, Robalo Nunes T, Sauvan V, Perez M, Graf C, et al. A serious game designed to promote safe behaviors among health care workers during the COVID-19 pandemic: Development of “Escape COVID-19”. JMIR Serious Games. 2020; 8: e24986. DOI: https://doi.org/10.2196/24986 

  40. Suppan L, Abbas M, Catho G, Stuby L, Regard S, Harbarth S, et al. Impact of a serious game on the intention to change infection prevention and control practices in nursing homes during the COVID-19 pandemic: Protocol for a web-based randomized controlled trial. JMIR Res Protoc. 2020; 9: e25595. DOI: https://doi.org/10.2196/25595 

  41. Loda T, Loffler T, Erschens R, Zipfel S, Herrmann-Werner A. Medical education in times of COVID-19: German students’ expectations - A cross-sectional study. PLoS One. 2020; 15: e0241660. DOI: https://doi.org/10.1371/journal.pone.0241660