Introduction

Medication errors may result in patient harm and is a top concern for patient safety, globally [1, 2]. These errors are generally due to failures in the medication use process, which include prescribing, clinical review, dispensing and medication administration errors. Becker’s Hospital Review had previously reported that medication errors during COVID-19 were due to hard-to-read medication labelling, lack of training in medicine barcodes, inaccurate weight-based dosing, wrong drug from automated dispensing, and missed doses due to personal protective equipment (PPE) rationing [3]. In fact, guidance for medication optimization and simplification due to COVID-19 had been issued from US, Australia, and international sources [4].

Healthcare practitioners and staff need to be competent in skillsets in medication safety practices. The World Health Organization’s Patient Safety Curriculum Guide [5] has suggested several approaches to teach and assess medication safety – including problem-based learning, case scenarios, role-plays and Objective Structured Clinical Examinations (OSCEs). Topics include prescribing, drug administration and calculations, identifying medication errors, contributing factors and prevention strategies, and performing medication/allergy history-taking, among others. In hospital institutions, medication safety competency training and assessments are generally conducted via e-learning modules, clinical case reviews, demonstration and practice using dummy mannequins, simulations with standardized patient actors or actual patients, and narratives/walkthroughs by the learner [6, 7, 8, 9, 10, 11]. COVID-19 has negatively impacted programs with on-the-job training, and hands-on components cannot be easily delivered through online programs due to social distancing, infection control and prevention measures. Clinical clerkships and attachments were either suspended, or face-to-face training of essential skills kept to small group sizes and repeated frequently, increasing time, effort and manpower [12]. Several challenges were amplified during COVID-19, which included high labor-intensiveness (additional workload and manpower shortage to deal with COVID-19, and a high burden on trainers due to the need for more frequent training sessions to smaller group sizes), and resource-intensiveness (depleting medical supplies/personal protective equipment through frequent use), thus leading to disruptions in workflow processes, difficulties in providing timely and personalized feedback to learners, and the potential for direct patient harm. These labor and resource challenges faced during COVID-19 are not sustainable long-term. Nevertheless, skills-based competency trainings and assessments, especially in regard to medication safety are critical, and need to be assessed to an acceptable practice standard in order not to compromise patient safety.

Virtual reality (VR) is an attractive way to enable such training and assessment. The use of VR in healthcare education is not a new idea. VR technology stimulates multiple senses through the use of interactive simulations to engage and immerse users in a computer-generated virtual environment that generates feelings similar to real world events and objects [13, 14]. Immersive VR, which is what is being considered in this context, usually involves the use of three-dimensional (3D) head-mounted displays and can elicit high levels of immersion and presence (i.e. the psychological illusion of being in the virtual environment) [14]. There are several advantages to the use of VR for healthcare education [15, 16], such as developing communication and higher-order skills (e.g. analyzing and problem-solving skills), enabling students to learn at their own time and pace, allowing students to put themselves in the shoes of the healthcare professional to develop the skills and comfort towards patient interactions, and it is ethically justifiable as it can minimize the risks of medical/medication errors and complications in real-life patients.

VR technology has been used in various healthcare settings for patient care, such as for physical and stroke rehabilitation [17, 18], cardiopulmonary resuscitation [19], cerebral palsy [20], pain and anxiety [21, 22, 23], pediatric oncology [14]; as well as in medical education to teach anatomy [24, 25], communication skills and empathy [26, 27], dental and surgical education [28, 29, 30], and nursing education [31, 32], among others. Applications in pharmacy include the teaching of content-based knowledge in the medical sciences (e.g. anatomy, physiology, pharmacology, pharmacotherapy) [33, 34, 35], health communication and compounding [36, 37], and as a method of lesson engagement in long distance problem-based learning and team-based learning [38]. In a meta-analysis comparing the pass rates of medical students undergoing VR training versus traditional modes of learning, the researchers reported that the VR group performed significantly better than the traditional education group (odds ratio, OR = 1.85, 95% confidence interval, 95% CI: 1.32–2.58) [39]. In their subgroup analysis, they also found that postgraduate (OR = 1.64, 95% CI: 1.13–2.39) and hospital residents (OR = 15.73, 95% CI: 2.35–105.04) in the VR group performed better than their respective counterparts in the traditional education group. The researchers suggested that VR could be used to train complex skills and specialized knowledge in the medical field. However, to our knowledge, VR technology has not been explored as part of on-the-job competency training and assessment for pharmacy staff and trainees, which plays a role towards practice licence registrations. The closest studies that we could find in the literature were the use of virtual patient avatars for pharmacist pre-registration training, but their content was mainly focused on improving clinical/pharmacotherapy knowledge [40, 41, 42], and not specifically catered towards medication safety. The use of virtual patient avatars has been suggested to be able to provide multi-sensory feedback for learners to learn the skills and applications needed in different healthcare contexts [43]. When virtual avatars are combined with social VR platforms (i.e. platforms that use VR for interaction, collaboration and interpersonal communication), it allows the conveyance of non-verbal cues (e.g. facial expressions, posture) that can lead to better coordination and understanding of the communication to users [44]. However, the users’ perceptions of the social interactions in these social VR platforms are limited by the fidelity of the avatar’s appearance and behavior.

In light of the disruptions in clinical clerkships and attachments and a rapid uptake of digital technologies in education during COVID-19, a VR application for training of medication safety practices was deemed to be attractive at our institution. Hence, as part of our institution’s quality improvement initiative, we decided to embark on a virtual apprenticeship program (called VRx) to train our pharmacy staff and trainees (i.e., pharmacists, pre-registration pharmacists, pharmacy technicians, pharmacy assistants, pharmaceutical science students on hospital attachments) on medication safety practices during pharmacy operations. Our primary objective is to create and develop a VR platform incorporating a virtual patient encounter (VPE) to train key tasks in 2 main operational areas:

1. Prescription (Rx) validation and processing – Checking for legality, authenticity and information completeness and accuracy of prescription/medication orders; identifying medication-related problems (e.g., inappropriate prescribing); performing pharmaceutical calculations and ensuring that drug label information are correct; and
2. Medication processing – Error-free picking, packing and labelling of medications; ensuring appropriate packaging according to safety, stability, patient and legal requirements (e.g., light-sensitive, temperature-sensitive, expiry, etc).

A secondary objective is to evaluate the usefulness, user-friendliness, and engagement of the VRx program through pilot testing of the VPE among pharmacy staff and trainees.

Methods

Development of the VRx program

The VRx program was designed to provide a form of virtual apprenticeship training to pharmacy staff and trainees at our hospital institution, in order to continue training and assessment of medication safety practices without compromising the COVID-19 regulations and restrictions for healthcare professionals and patients. This program was developed in a 3D virtual environment using the Unity 3D engine (Unity Technologies, CA, USA). A series of key competency tasks based on the healthcare professional competency standards set by the Ministry of Health, Singapore, was built around a virtual patient encounter (VPE) in an outpatient pharmacy. These competency tasks included (Figure 1): (i) validating and processing prescriptions, (ii) checking the accuracy of medication orders, (iii) performing pharmaceutical calculations, (iv) processing medications and labels, (v) preparation of medications, and (vi) picking, packing and dispensing medications. The storyboards of the scenarios were developed using Microsoft Excel and a VR vendor was engaged to develop the virtual environment and VPE through the institution’s procurement process. The main technologies used were a Razer Blade 15 laptop and an Oculus Rift S (Facebook Technologies LLC, USA), which comprised of a 3D head-mounted display/headset worn by users to move around in the immersive virtual pharmacy environment and hand-held controllers to perform the key tasks. The graphics were bought from the Unity engine’s asset store or designed using 3DS Maya (Autodesk Inc., CA, USA) and Blender (Blender Foundation, Amsterdam, Netherlands). The backend database was developed using an open-source software (Learning Locker, Learning Pool, USA). The player’s responses and performance in the tasks were recorded real-time via a training dashboard, which could then be used for review and feedback. Programming scripts used to develop the virtual environment, VPE and training dashboard included a combination of python, C++ and javascripts.

Figure 1 

The key competency tasks incorporated in the virtual patient encounter as part of the VRx virtual apprenticeship program.

Results

Qualitative feedback

Qualitative feedback indicated that the strengths of the VRx program were in relation to providing an authentic learning experience for trainees (Figure 2a). Besides providing a realistic simulation of real-life scenarios and work processes that was interactive, fun and engaging to the participants, VRx allowed them to have a hands-on experience in a safe environment without the stress of catering towards the pressures in a physical pharmacy. The program was not only able to help the participants reinforce their knowledge and concepts, and familiarize with the picking and packing process, but it was also able to help minimize disruptions to the workplace and workflow processes. Through immediate feedback and a more structured way of training, the participants reflected that they were more aware of, and able to learn from their mistakes on-the-spot, as well as found it easier to understand, remember and apply the concepts to their work.

Figure 2 

Word clouds from participant feedback reflecting the strengths, difficulties faced and suggestions for improvement of the VRx training program.

In terms of difficulties faced during the sessions, many of them found difficulty in navigating the virtual environment and scenarios as they were not familiar with the functionalities of the hand-held controllers and buttons (Figure 2b). They felt that they needed time and practice to adjust and adapt to the controls. In addition, some of them felt that the VR headset was heavy and that their line of vision was not optimal due to their height, which caused strain on the eyes and neck. Wearing the headset for long periods of time also caused dizziness and nausea in some participants. Furthermore, some participants felt that some parts of the instructions were unclear or vague, which resulted in them not knowing what to do in certain scenarios.

Suggestions for improvements were mainly in relation to the technological aspects, as well as on the training (Figure 2c). Technological enhancements included refining the sensitivity, use and functionalities of the controllers, providing a help screen for users to look up the controller functions, use of the gaze function to transition between the scenarios; as well as improving the resolution of the images and fonts. In regard to the training sessions, it was suggested that these sessions could be broken down into smaller segments of shorter durations, as well as adding more training scenarios to include a wider variety of drugs with different methods of packing. The participants also suggested to reduce some of the repetitive steps in picking and packing, skip some of the intuitive steps (e.g. opening and closing of bottles) and shorten the time needed to transition between steps in the tasks. Other suggestions included having a hybrid mode of both VR and real-life hands-on activities, simplifying the instructions, providing a briefing to the VR technology and controllers, tailoring some tasks to specific departments and the need to consolidate the learning for trainees at the end of the program.

Discussion

An in-house virtual apprenticeship program called VRx has been developed as part of our institution’s medication safety training. Based on the healthcare professional competency standards set by the Ministry of Health, Singapore, the VRx program focuses on competency skills training to complement on-the-job training for pharmacy staff and trainees during the COVID-19 pandemic. The activities in VRx targeted 6 medication safety domains according to the Pharmaceutical Care Network Europe guidelines (Table 3) [46]. These activities included preventing potential problems with the lack of pharmacotherapy effects on treatment effectiveness; preventing treatment safety issues resulting from missing/incomplete drug information; identification of drug-related problems related to the drug (e.g. selection of the drug, drug form, dose schedule and treatment duration); and identification of drug-related problems related to the logistics of the prescribing and dispensing process (e.g. identifying errors and legal issues with the prescription and perform an intervention where appropriate; and affixing appropriate auxiliary medication labels so that drug effectiveness is not compromised), among others. The advantage of this program is that the VR component is able to provide an authentic and safe environment for staff and trainees to learn about medication safety practices through responsible decision-making, but yet can still learn from their mistakes without experiencing the severe negative consequences that may occur in real-life patient care. Specifically, staff and trainees would be provided with immediate feedback on their mistakes, who would then need to acknowledge in order to proceed with the next step or stage.

The design and development of VRx is based on several pedagogies in simulation and healthcare education. Recent systematic reviews have reported that ~70% of studies have no learning theory as a foundation of the VR interventions [47, 48]. Our project adapts Zhou’s VR Interaction Learning Model [49] (Figure 3) by merging the features of VR-supported technologies with pedagogical frameworks in VR simulations and healthcare education to design and create our VPE scenarios. There are 2 consideration factors when designing the VPE scenarios in order to bring about the intended learning benefits – VR immersion and VR roles. In the VRx program, a semi-immersive desktop VR is used by the trainer to observe how the trainee completes the scenarios, while the trainee undergoes a fully immersive VR experience in the virtual environment, using the VR headset and controllers. On the other hand, the virtual environment is designed to mimic the workplace so that trainees are able to apply and practise their medication safety knowledge. However, unlike actual on-the-job placements which are more unstructured, the competency skills that are needed by the trainee are made explicit as demonstrable task elements in order to scaffold the trainee’s learning in a more structured and obvious manner. The high proportion of participant responses agreeing that VRx helped in prescription processing skills and the qualitative feedback indicate that we are on the right track.

The simulation-based pedagogy that we used to develop VRx was according to de Freitas’ Four-Dimensional Framework (Figure 4) [50], which provided consideration factors in terms of the context of the application, learner specification, mode of representation and pedagogic considerations. Our target population was more clearly defined through the learner specification framework (i.e. students, graduates and pre-registration pharmacists attached to our hospital, polytechnic graduates from pharmacy/ pharmaceutical science specializations, and new hires in our department). Although the levels of technological and manpower resources needed were low to medium (i.e. laptops with VR headset and controllers, and only one trainer needed per session), the levels of fidelity, realism and immersion needed to be medium to high, so that the trainees would be able to apply their knowledge and skills to realistic situations in the VPE and their skillsets could be transferrable to the actual workplace. In this respect, the virtual environment was designed after our hospital’s outpatient pharmacy (prescription and dispensing counters), medication storage and preparation areas (e.g. medication shelves and bins, refrigerators) and compounding areas (for compounding, preparation and labelling of non-sterile products). However, we were conscious that the virtual environment also needed to be generic, so that it could be transferrable to the pharmacy facilities in other hospitals. As such, our virtual pharmacy also adhered to the minimum standard guidelines for pharmacy facilities advocated by the American Society of Health-System Pharmacists [51].

The learning activities that were designed in the VPE were aligned to the competency learning outcomes of processing prescription and medication orders, and medication safety practices, which required the trainees to hone their skills in clinical reasoning and decision-making, effective communication, clinical documentation and organization, patient history-taking and patient education, among others. Certain cognitive elements, such as prescription processing and validation, and dose calculations, can be supported by good decision support systems for prescriptions and medication orders so as to enhance medication safety. However, there are limitations in practice settings. For example, the risks of medication errors can still occur due to issues such as missing or incorrect information on the prescription [52, 53] (e.g. missing drug administration routes or incorrect drug forms), wrong calculations, transcriptions and/or correlations of medication doses to their corresponding drug strengths and drug forms [52, 53] (e.g. converting insulin dose units to the number of insulin pens based on type of insulin prescribed), and with extemporaneously compounded medication products [54, 55, 56] (e.g. medications for patients with difficulty swallowing, reconstitution/dilution of injectable medications, pediatric doses of formulations that are not available commercially). Furthermore, clinicians tend to turn off the alert prompts in their clinical decision support systems due to alert fatigue [57], which in turn also impacts medication safety. The whole medication dispensing process depends greatly on behavioral input, whether it be assessing the appropriateness of prescriptions, interpreting prescriptions accurately, finding the appropriate medications to pick, pack and label, or making sure that the medication order is correctly fulfilled. Safe dispensing practices can be supported through highly motivated individuals with a good understanding of medications and dispensing equipment, working in a well-coordinated team [58]. On the other hand, being unaware of or under-estimating particular risks, such as not factoring in shelf lives/expiry dates of drugs; assuming certain drug administration routes/drug forms (e.g. intravenous versus intramuscular injections, or eye-drops versus ear-drops); negligence or lack of attentiveness; fatigue and stress; misinterpreting drug information (e.g. look-alike-sound-alike drugs); inappropriate/unclear communication or documentation; or being governed by inadequate guidelines and/or unclear organizational routines can also predispose to unsafe dispensing practices [58, 59]. As pharmacists are the last line of defence before the patients receive their medications, these skills still need to be instilled in pharmacy staff to ensure that medication errors are minimized throughout the whole medication dispensing process. As such, the team deemed VR to be a suitable technological method of training instruction. Furthermore, it was deemed that face-to-face briefing and debriefing by both the trainer and the trainee’s preceptor would also be needed to reinforce the content learnt through VRx. From the participants’ feedback, we intend to incorporate technical briefings on how to use the VR technologies before the trainees undergo the VPE in the future, so that they can have a more seamless and enjoyable experience.

The Miller’s Pyramid of Clinical Competency framework [60, 61] was used to provide guidance on designing our scenarios to train and assess the trainees’ skillsets (Figure 5). Embedded within the VPE scenarios were different variations of multiple-choice questions (MCQs) and short answer questions for trainees to answer in order to reinforce their knowledge during the training sessions, but could also double-up as quick assessments of the trainee’s knowledge. The VPE targets the skills competency layer of the Miller’s Pyramid by providing a virtual simulation of a standardized patient encounter, so that trainees can apply their knowledge and skills into specific demonstrable tasks that indicate their competencies and practice readiness, in the form of entrustable professional activities (EPAs) [62]. For the EPA of dispensing of medications (Figure 5), the domains of competence (knowledge, skills, attitudes and experiences) that the trainees were required to exhibit were translated into ten competencies (C1-C10 of EPA framework) which they had to be proficient in, and 11 milestones (M1-M11 of EPA framework) which they had to demonstrate. The trainees’ competency in medications dispensing would then be assessed and rated using five progressive levels of learner behavior from pre-entrustable to entrustable. As suggested in the participant feedback, we intend to incorporate a wider variety of VPEs which contain a variation of drugs and different methods of packing, so as to train a broader set of competencies through EPAs.

Our project was aimed at improving the knowledge and competency skillsets in medication safety practices among healthcare professionals, with a beginning focus on pharmacy staff and trainees. By providing a strong base and focus on medication safety through VRx, we aimed to put “Patients at the Heart of All We Do” (part of our organization’s mission), so that our staff would be acknowledged as trusted healthcare professionals who would be able to provide quality services for our patients. In the development of VRx, we wanted to be able to harness appropriate, evidence-based technologies, and not just use the latest technological trends, to train the newer generations of pharmacy professionals. In addition to enhancing engagement and motivation of our staff and trainees through the technology, we wanted VRx to play a dual role of enhancing their digital literacy through the simulation experience. Some of the participants were surprised that our institution’s training program had incorporated such “sophisticated technologies”. Amidst facing some technical difficulties and kinks, the participants were still excited to undergo the VRx experience, as many of them did not have the opportunity to to experience their learning in a virtual simulated environment in pharmacy school. Moreover, when this program was presented to pharmacy managers at the other hospital institutions in SingHealth (one of three integrated healthcare clusters in Singapore’s public healthcare system) [63], they were also inspired to consider adoption of such innovative simulation solutions to train and assess their staff and trainees. Besides the use of a Windows i7 laptop that is compatible with the Oculus VR equipment, VRx can be adopted and incorporated as part of their training programs.

In terms of sustainability, VRx was postulated to be able to decrease the manpower needed for training in the medication safety curriculum. According to anecdotal feedback from some senior pharmacists who tried playing VRx, they shared that VRx could potentially reduce the number of man-hours in their training programs. As the VPE scenarios were repeatable and could provide a standardized way of training and assessing the individual, the pharmacy managers also felt that the VRx training could potentially improve the quality of their medication safety training and assessment programs as well. Additionally, they suggested that the VPEs could potentially be useful to teach about rare diseases and/or rare medical or medication events that the trainee might encounter in real-life practices. As such, our team has been engaged as consultants by pharmacy educators from other healthcare institutions to help them develop similar training initiatives through VR.

Limitations and Future Work

Besides the difficulties faced by participants during the training sessions and their suggestions for improvement, we faced challenges in terms of booking of room facilities for the training sessions within our hospital institution, due to repurposing of the rooms to cater for COVID-19 isolation and quarantine requirements. In addition, as VRx was a prototype, we only had one set of equipment (VR headset, controllers and laptop) to conduct the training sessions, which limited the number of participants who could be trained within a certain time period. The equipment had to be sanitized and wiped down after each use to reduce the probability of COVID-19 infections, thus each training session could only be conducted at least 30 minutes apart.

Participants had to take short breaks between the stages to prevent eye strain. Those who suffered from severe motion sickness were not able to attend the VRx program. For those who had just graduated and did not have any working experience, they had difficulty applying what they had learnt in school to the VPE scenarios, which then required the trainer to explain and jog their memory to recall the concepts needed to complete the tasks, thus resulting in more time spent in the sessions than what was originally allocated. Due to the mask-wearing requirement at the workplace, participants had to spend additional time readjusting their masks, spectacles and/or VR headsets in order to avoid fogging of the lenses. Moreover, some of the participants required frequent reminders on the functionalities of the controllers in order to proceed with the tasks. As such, we are now in the process of seeking external funding to develop more scenarios, modify the prototype and obtain more equipment. In addition, we are developing some pre-training material on our internal e-learning platform for trainees to prepare/revise their content knowledge before participating in the VRx program.

Lastly, we did not manage to evaluate the competencies of the trainees due to the additional workload of staff and re-deployment of manpower to deal with COVID-19 at our hospital. However, we acknowledge that the evaluation of trainees’ competencies is important to identify whether the learning outcomes were achieved. Therefore, we are currently developing a post-quiz to be completed by future trainees at the end of the virtual apprenticeship program, so as to consolidate the knowledge for the trainees, identify their areas of weakness and provide a more structured and personalized feedback to them. In addition, a comparison between two cohorts – one undergoing VRx and another undergoing the traditional on-the-job training without VRx – is intended to assess whether VRx has improved the competencies of the trainees at our institution.

Conclusion

A 3D virtual reality program (called VRx) has been developed to train pharmacy staff and trainees on medication safety practices. During the training sessions, staff/trainees are immersed in a virtual pharmacy environment, and they have to complete several key tasks to hone their competency skills in medication safety via visual/motion capture technologies. Overall, pharmacy staff and trainees agreed that VRx was able to provide a realistic, engaging and interactive experience to acquire the required proficiencies in medication safety practices, hence it is intended that VRx be used as a form of virtual apprenticeship in our institution’s training program. The competencies in medication safety are generic and thus can be applied, adapted and scaled-up for training of other healthcare professions. With the global emphasis on interprofessional education, we envision that VR training will attract the medication safety curricula of many universities and training/vocational institutes in the near future.