A mixed reality telemedicine system for collaborative ultrasound diagnostics and ultrasound-guided interventions

Introduction

Since its introduction in the 1950s ultrasound (US) has been the most widely used imaging modality in the world and has application in nearly every medical specialty. But US examinations and especially US-guided interventions are hard to learn and need much training to perform at acceptable standards. This is a real challenge because, on the one hand, the training opportunities are sparse, and on the other hand, they imply practicing on real patients. Moreover, in most countries, the US training is not part of the standard medical curriculum. Therefore, most practitioners are looking for alternative education opportunities, but these are quite expensive and when they are booked they can easily be cancelled due to COVID-19. As a result, second opinions are a daily business in nearly every hospital.

For over 10 years, emergency rooms (ERs) across the European Union (EU) have been overcrowded and often lack experienced staff members (1). As published by the European Regional Office of the World Health Organization (WHO), EUs’ ER staff has to treat more than 100 million patients every year; often because of insufficient primary care (2).

Concept and methods

To solve or minimize the effects of the listed problems we designed a tele-assistant tool that will allow expert physicians to support less experienced physicians in the emergency room during diagnostic procedures like FAST and interventions like the placement of CVCs.

The ER physician performs the US in the emergency room while wearing an augmented reality (AR) headset (HoloLens 2). The emergency room is also equipped with a 3D scanner to create a 3D point cloud of the US examination.

The physician can see the US image projected on the patient through the HoloLens glasses, making it easier to focus on that area during the examination. S/he does not have to look at the US device monitor.

If the examining physician is uncertain about making a diagnosis, our system can be used to connect to a remote expert physician, to ask his/her opinion. The expert wears a virtual reality (VR) headset (e.g., HTC Vive) through which s/he can see the US image as well as the 3D point cloud of the patient and the ER. With the help of the VR glasses controller, the expert can place a virtual US transducer in the shared virtual space, therefore s/he can give advice and show how to scan correctly. With the use of HoloLens we could visualize the virtual transducer in the correct location on the patient, allowing the doctor in the emergency room to see the position suggested by the expert as shown in Figure 1. The expert can also connect to the HoloLens camera so that the examiner can see “through the examiner’s eyes.” They can also communicate via the HoloLens microphone and speaker (Fig. 2).

This method saves time as expert opinion can be obtained more quickly. It can take pressure off the shoulders of healthcare workers, reducing the chance of errors and resulting in fewer patients being mistreated. This also significantly reduces costs.

Our main technical objectives are to create a combined AR/VR scene and interaction platform with the following features:

• examiners can use AR glasses to have a real-time AR view of the US image and the patient;

• experts can use VR glasses to “see through the eyes” of the examiner;

• experts can create a virtual transducer that can be blended into the AR view of the assistant; to show assistants where to place the transducer and in which angle;

• both can talk to each other.

System components

The proposed system consists of multiple components and plug-ins connected as seen in Figure 3. In the next subsections, more details about these components are provided.

HoloLens 2

Microsoft HoloLens 2 is a headset equipped with see-through holographic lenses and several different sensors and cameras, allowing a combination of real images and computer-generated images in the same field of view. Thanks to gesture, eye, and voice control capabilities and wireless technology, the device can be used hands-free and does not impede free movement. This can be particularly important in medical applications, where doctors need their hands to perform their tasks. This is why HoloLens is also used in other medical applications, including during surgery.

With HoloLens 2 it is possible to visualize the US video stream on the patient, just over the area of the body that is being scanned. Therefore the physician doesn’t have to look away at the US device’s screen, allowing them to focus on the US image while looking at the patient. The virtual US transducer controlled by the expert (apparent in the same field of view) will show the suggested position, improving the US image.

3D scanner device: Azure Kinect

Azure Kinect is a Kinect sensor made by Microsoft. It is equipped with a 12-megapixel RGB camera, a 1-megapixel depth camera. The device can also connect to Microsoft Azure cloud services, opening up a wealth of possibilities.

In the emergency room, a 3D scanner records what happens during the US examination. The data is processed later after it has been transmitted to the expert doctor’s site, then the 3D point cloud is created. Azure Kinect is the right tool for this, as its streaming capability makes it feasible.

ApertusCore

ApertusCore is a programming library, developed by SZTAKI, written in C++11, with the following features: modular, embeddable, platform-independent, and easily configurable. It contains basic software interfaces and modules for logging, event-handling, and loading plug-ins and configurations (9). ApertusCore is where the various plug-ins are connected. The streamed data from the 3D scanner is managed by ApertusCore, and the HoloLens and HTC Vive glasses are connected through it in the shared virtual space. The US data streamed through SonoXR is also fed into and connected to the virtual space.

ApertusVR is a plug-in connected to the ApertusCore that is responsible for the visualization, this allows us to create a shared VR world scene where the point cloud, the US video, and the other data are visualized.

As the ApertusCore platform is independent it is ideal to connect multiple devices and facilitate communication. ApertusVR can be used to create a shared virtual space for both HTC Vive and HoloLens VR glasses.

SonoXR

In most diagnostic cases and interventions the best practice for physicians would be to have a straight look into the human body. But current imaging modalities like computer tomography, magnetic resonance tomography, or US only provide an indirect look. This indirect look is a major challenge during diagnostic procedures and a real problem during interventions.

With the assistance of SonoXR, physicians can display the live US images through AR glasses (like HoloLens 2) at the point the image data are taken. This gives US users a kind of “superhero X-ray vision” that helps them see anatomical structures where they actually are. In addition, a virtual US display can be freely positioned in space to increase the ergonomics of the intervention. US-AR-based interventions are faster, less sensitive to errors, more efficient, and, as a result, reduce the risk of infections (10,11). An example of the view through the HoloLens 2 is shown in Figure 4 (with the use of a ”phantom”).

SonoXR consists of the

• SonoXR-server, running on a mini-PC (SonoXR transmission box; Windows 10) and

• SonoXR-client that runs on a HoloLens 2 (Windows Holographic)

The SonoXR-server collects the US screen image data in real time via an HDMI connection from a US scanner with 25-50 frames per second and up to 1920 × 1080 px. After cutting the US screen image data to the pure US image data content the final images are sent wireless to the SonoXR-client on the HoloLens 2.

On the HoloLens 2, the US images are shown

• directly beneath the US transducer to realize the above-mentioned “superhero X-ray vision”

• on a so-called “virtual screen” that can be placed by voice commands to any position in space the user wants to. Additionally, the virtual screen can be zoomed, rotated and basic image processing methods like changing brightness and saturation can be performed

Discussion

The designed tool using VR and mixed reality (MR) to visualize US images offers essential features that can resolve some of the issues encountered in ER departments across Europe. First, it allows to provide an answer to the recurrent problem of the need for experts’ opinions, as experts can be called in, they can visualize the images and the surroundings, and provide immediate feedback. This happens through the use of SonoXR technology who projects the US images on a virtual headset. Furthermore, an Azure Kinect 3D scanner provides an overview of the entire scene. Secondly, it can solve the issues of the scarcity of US training, as it can be used for remote learning, in a bidirectional manner. This means that on the one hand, the expert will help to improve the handling of the US probe by the less experienced user, with the use of his virtual US probe. Thanks to the HoloLens, the physician will see the proposed position of the probe on the patient. And on the other hand, it can also be used the other way around, namely, that the expert will give live training sessions to many people in remote locations by streaming his images to the VR headsets of the trainees.

However, despite the evident advantage of such an innovative way of training and collaboration, the essential “clinical proof” has to be provided for every planned use case and practitioners have to get used to the new tool. Therefore we plan to set up clinical trials to prove efficacy of the system. We focus on a “plug and play” approach by the physicians. The system must

• be quick and easy to set up: Physicians in the ER do not have the time to start and use complicated systems.

• have an intuitive user interface: The user interface needs to be so intuitive that even a physician who does not use the system every day will be able to use it.

• present durable and stable hardware: Due to fast and patient-focused work, the equipment in an ER has to be robust. The AR glasses are the critical point in the setup.

• have a low latency: The expert physician needs to see the ER and hear the assistant physician in real time.

Conclusions

In summary, we propose an innovative way to deal with the shortage of US experience throughout the ER departments, by using AR and VR, a 3D Azure Kinect scanner and the SonoXR technology. The system can provide a fast and low-cost way to improve healthcare around the world but the different possible use-cases still need the (clinical) validation.

Disclosures

Conflict of interest: The authors declare that they have no conflict of interest.

Financial support: The work described here was partly funded by the German Federal Ministry of Education and Research in the project “ARNI” (Augmented Reality Needle Intervention; funding reference: 16SV8152).