Validation of a Cognitive Task Simulation and Rehearsal Tool for Open Carpal Tunnel Release

Article information

Abstract

INTRODUCTION

The training of future surgeons is a lengthy endeavor, requiring substantial investment by medical schools and residency programs. It is further complicated by the ethical and safety considerations involved in novice surgeons operating on patients. This concern was cogently summarized in a surgical journal many decades ago-“a first experience has no place in the operating room” [1]-but we still find this to be the case for many residents. At the same time, residency programs are also now faced with mandated duty-hour limitations, restricting how much time a trainee is allowed to spend in the hospital. For surgical residents, this effectively means a prescribed and finite amount of time that can be spent in the operating room [2]. For both of these reasons, it is increasingly important to maximize preoperative learning and postoperative knowledge consolidation, as well as to provide educational tools that can be utilized outside of the hospital to allow surgical training to thrive.

Many institutions have begun to incorporate surgical simulations in an effort to expand their training curriculum beyond what is provided by random opportunity [3]. Early versions of these simulators were created to provide a user-guided 3-dimensional environment to explore the steps of an operation. They offer a type of rehearsal prior to encountering a real patient [4]. Some are quite elaborate and require a significant investment of time, space, and funding by the sponsoring department and are limited to one or few procedures. Many of these simulators are predominantly focused on teaching technical skills, with little focus on intraoperative decision-making.

In an effort to counteract some of the costs associated with simulations, many programs now use computer modules as a way to augment learning [5]. Our group, for example, created an online microsurgery curriculum that we have previously validated as a cheap and efficient way to provide complex surgical skill training to our plastic surgery residents [6]. As mobile technology becomes increasingly ubiquitous and powerful, it is now possible to create high-fidelity apps that provide true on-the-go learning for trainees of all stages. It is clear that current trainees are becoming increasingly comfortable with incorporating smartphones into their clinical practice, with as many as 55% of Accreditation Council for Graduate Medical Education trainees requesting app-based reference materials [7].

In this study, we assessed one such smartphone application, Touch Surgery app (Kinosis Ltd., London, UK). This platform may be described as a cognitive-task surgical simulation and rehearsal tool. In these simulations, operative decisions are mapped to a virtual 3-dimensional patient, allowing the users to review the steps of a surgical procedure. The user progresses through the surgical procedure by interacting with the virtual patient through every step of surgery (Fig. 1) and is required to answer multiple-choice questions. At the end of the simulation, an inbuilt assessment tool (Fig. 2) evaluates cognitive competence and maps an individual's learning curve. The aim of this study was to evaluate the construct, content, and face validity of the open carpal tunnel release surgical procedure on the Touch Surgery app.

Fig. 1

Screen shot from the simulation app

This screen shot shows a representative image of the smartphone-based surgical simulator. This phase of the simulation guides the user through the markings for an open carpal tunnel release. The trainee must utilize the touch screen to draw over the palm similarly to how they would carry out the procedure in the operating room.

Fig. 2

Built-in assessment tool

This is a screen shot from the built-in assessment engine within the simulation app. It is able to show users their most recent simulation score as well as to track their progress as they improve their surgical skills and technique.

METHODS

RESULTS

A total of 30 participants volunteered to participate in the study. Twenty of the participants were novices and 10 participants were experts.

For construct validation, the expert cohort scored significantly higher than the student cohort, with a median of 74% versus 45%, respectively (P<0.0001) (Fig. 3).

Fig. 3

Box plot demonstrating construct validity

Construct validity is the ability of a tool to demonstrate differences in skill levels. This box plot of the raw scores that participants received on the simulation shows a statistically significant difference between experts and novices, confirming construct validity.

On the feedback questionnaire, scores with a median of 4 or greater were reported for all statements regarding content (Table 1), face (Table 2), and acceptability (Table 3).

Table 1

Content validity

Table 2

Face validity

Table 3

Acceptability validation

DISCUSSION

The results of this study demonstrated that the Touch Surgery app has construct, face, and content validity for carpal tunnel release surgery. Construct validation corresponds to a statistically significant difference between novices and experts, which provides an evidence base for the scoring algorithm of the app. We believe that differences in techniques and/or the use of alternative instruments ultimately led to variability in the experts' score and the median expert score of 74%, as opposed to a gap in the knowledge of the experts.

Face and content validation jointly signify that a simulation teaches what it intends to teach and resembles the real world.

The plastic surgery milestone project assessment tool by the Accreditation Council for Graduate Medical Education and the American Board of Plastic Surgery lists surgical skills evaluation as one of the 6 components in accreditation. Items within this evaluation rubric pay particular attention to surgical marking, positioning, preparation, draping, surgical steps and flow of the operation [10], all of which are objectively assessed within the app. This is not to be confused with the clinical competence to perform the procedure, but could help trainees reach clinical milestones faster. Technical skills training still needs to be covered in the surgical curriculum, as well as observing a varied mix of cases and presentations.

Tremendous interest has emerged in simulations as a means to enhance medical education for a variety of reasons, including duty-hour restrictions and ethical concerns over the use of patients for training [11]. The Touch Surgery app provides residency programs with the ability to have residents simulate and safely rehearse surgical procedures outside of duty-hour restrictions through a simulation tool that is available on smartphones.

The use of cognitive task analysis in surgical training has been shown to accelerate skill acquisition and to improve performance in procedures such as central venous catheterization and laparoscopic cholecystectomy [1213]. It has also been demonstrated that in some cases, cognitive task analysis may be as effective or more effective than on-the-job training [14].

This app can also provide residency programs with objective data that could be used to measure a resident's cognitive knowledge of a procedure, meaning that institutions that adopt this app will be early movers into a competency-based curriculum. Moving towards curriculum integration is necessary to the ensure effective incorporation of simulations within a residency program, as the voluntary use of simulation tools has been shown to lead to minimal participation [1516]. A consensus also exists among the majority of junior residents that simulation should be made a mandatory component of training [1617].

We strongly believe that apps have the potential to be valuable additions to surgical learning for trainees of all levels as part of a training curriculum. The beauty of low-cost and easily delivered app-based content derives from the unfettered access to such content.

This study evaluated participants simulating a single procedure at one institution. Future work will expand to include multiple centers and multiple procedures. We aim to recruit an intermediate group and assess their scores. More importantly, as the validity of Touch Surgery app has been demonstrated, we can now begin to deploy the platform within the curriculum and observe the learning curves of residents and the transfer-ability of skills. At present, the platform only includes a single approach to carpal tunnel release, using our open technique. Further work needs to be done to demonstrate other open and minimally invasive techniques. It may also be feasible to use other instruments to perform the same objectives, and this should be incorporated into the simulations.

Touch Surgery app was demonstrated to have construct validity, with statistically significant differences in results observed between experts and novices. It has also demonstrated face and content validity based on ratings by our expert groups. This application has the potential to improve access to surgical education around the world.

Article information Continued