HoloMed – How Augmented Reality supports during surgeries

How augmented reality makes the invisible visible

Augmented reality (AR) is making its way into the operating room: Whenever surgeons wear the HoloLens AR glasses during punctures of the cerebral ventricle, the navigation aid is displayed directly into their field of vision. We are currently working on the user-friendly design of this novelty together with our partners in the HoloMed research project. An initial test run at Günzburg hospital has now been completed successfully.

SAFELY ON TARGET WITH AUGMENTED REALITYVentricular punctures are among the most common routine interventions in neurosurgery. Nevertheless, the catheter that is inserted into the brain is only optimally placed in two thirds of the cases. The problem: The surgeon does not see the target—a cavity in the brain filled with liquor.

In future, the HoloLens will provide support in this procedure: Using CT or MRT data of the patient, HoloMed computes an exact 3D model of the brain before surgery.

The HoloLens projects the model with millimeter precision onto the patient’s head, making the invisible visible.
Moreover, graphical elements help the surgeon to find the right angle and position. The benefits are obvious: a lower error rate and more safety for patients and surgeons.

FORGET ABOUT THE USER – FORGET ABOUT US!

We entered the relatively new field of augmented reality in surgery via an extensive analysis of the context of use. Several visits to medical facilities provided insights into the target group and helped us create life-like personas. Using storyboards, we documented possible procedures and design solutions for the AR application. An online survey among surgeons revealed their stance towards the use of AR.

TEST RUNS PROVIDE INSIGHTS FOR UX

Together with our partners, a functional prototype was created. Whether this prototype will meet the high expectations in a medical environment will be verified in iterative tests under real-life conditions. At Günzburg hospital, we watched the surgeons during the entire test run, gaining valuable feedback. In general, the users at the hospital are very impressed by HoloMed: A CT scan showed that the catheter was perfectly positioned after the punction, the application is intuitive and runs smoothly.

Before it reaches marketability and perfect user experience (UX) though, we still have a couple of to-dos on our list: The virtual projection has to be perfectly aligned with the patient’s position. In order to achieve this, we use a system of markers that still has room for improvement. What is especially interesting for us to see is how the doctors perceive the holograms in the surgical site. Currently, the lighting conditions in the surgery room greatly influence the holograms’ visibility. And we have learned that some of them obstruct the visibility of the actual medical instruments. Moreover, we are taking existing concerns about hygiene and time requirements seriously.

FROM CONCEPT TO DESIGN

What should HoloMed's user interface (UI) look like? How can the individual work steps be clearly illustrated for the neurosurgeon? We got together in a creative circle and developed a consistent design language with a high recall value that combines functionality with aesthetics. In order to test the findings, we held a prototyping hackathon with the entire team of UX experts, designers and developers. This quickly helped us find out what works well and where there is room for improvement. And it gave us the opportunity to try out the application ourselves.

MAY WE INTRODUCE THE NEW HOLOMED UIWhen the neurosurgeon puts on the HoloLens and starts HoloMed, four menu items are displayed: preparation, Kocher’s point, incision and angle. The order matches the workflow of a ventricular punction. With the MRT & CT Scans button, the surgeons can interactively display any important scans of the patient.
Some holograms for navigation support are animated and pulsate so that they don’t constantly cover the working area and that the doctor can perfectly align the instruments with the visualizations.
HoloMed can be controlled simply via “long gaze”, i.e. by focusing the hologram for a certain period of time, and speech, allowing for hands-free operation during the complex procedure. A welcome side-effect: There are no problems with regard to the high hygiene standards in the surgery room.

PROJECT CERTIFICATION WITHIN REACH
Suitability for use, safety, and hygiene are also important factors for a successful product certification process. We are supporting the research group with our expertise in compiling Usability Engineering Files to pave the way for the product certification of the HoloLens.

MORE INFORMATION ABOUT HOLOMED
The research project “HoloMed – Context-sensitive support for surgeons in the operating room using augmented reality” (funding code 01IS17005) is co-financed by the Federal Ministry of Education and Research and carried out by the following project partners: mbits imaging GmbH, User Interface Design GmbH, Neurochirurgische Klinik der Universität Ulm (neurosurgical clinic of Ulm University) and KIT – Intelligente Prozessautomation und Robotik (intelligent process automation and robotics) (IAR-IPR). It runs until the end of 2019.

TEAM

Dominik Zenth

Kontakt

User Interface Design GmbH
www.uid.com

UIDlabs
Brandenburgische Strasse 27
10707 Berlin
Deutschland