System for Telementoring with Augmented Reality - STAR project

Telementoring provides procedural guidance of a trainee surgeon by an expert surgeon from afar using telecommunication. The System for Telementoring with Augmented Reality (STAR) is an innovative platform that relies on augmented reality technology (Hololens headset and tablets) and touchscreen displays to increase the quality of the communication between mentor and trainee.

The latest version of the STAR system is based the Augmented Reality Head Mounted Display (ARHMD) Microsoft Hololens. This version of the STAR system is comprised of two parts: On the mentor side, a large touch-screen display provides a view of the trainee's surgical field, while serving as an interface for the mentor to send visual annotations to the trainee. On the trainee side, the Microsoft Hololens allows the seamless augmentation of the trainee's natural view of the surgical field with annotations and illustrations of a surgical procedure provided by the mentor. In addition, the view of the mentor is augmented with the patient's vital signs and ultrasound images from a portable ultrasound operated by the trainee on-site.

As part of my work in the STAR project, I have worked on the development and validation of the STAR ARHMD system. I created the experimental design for the validation of the STAR system in a simulated austere setting. For the validation of the system, we conducted a user study with twenty US Navy corpsmen comparing our telementoring system (ARHMD) against an audio-only telementoring scenario. Two remote surgeons guided each corpsman through two cricothyroidotomies in a simulated austere scenario at the Naval Medical Center in Portsmouth Virginia. Expert surgeons performed an on-site evaluation of the participants using military criteria for successful cricothyroidotomy performance. Our poster about this work has been accepted to the  MHSRS 2019.

As part of the development of the STAR (ARHMD) system, I integrated the use of a handheld ultrasound device with the STAR mentor system. I created a virtual camera application to connect the portable ultrasound Ext-MicrUS to the mentor system using the WebRTC protocol. The ultrasound is used on-site by the trainee on a portable computer and the image of the ultrasound is streamed to the mentor so that he can provide assistance during treatment. In addition, ultrasound probe estimation was desired to augment the view of the mentor with information of the ultrasound image and the position and orientation of the ultrasound probe. For this purpose, I designed a 3D printed dodecahedron tracker that uses 10 fiducial trackers to allow for pose detection of the probe with respect to the HoloLens camera, such geometry provides the view of at least 3 fiducial trackers at any given time, generating reliable tracking.