A mixed reality telemedicine system for collaborative ultrasound diagnostics and ultrasound-guided interventions


  • Stefan Maas SomaView GmbH, Research and Development, Bochum - Germany https://orcid.org/0000-0002-5245-8203
  • Peter Kopacsi Institute for Computer Science and Control (SZTAKI), Eötvös Loránd Research Network (ELKH), Machine Perception Research Laboratory, Group of Shared Virtual Reality, Budapest - Hungary
  • Peter Kovacs Institute for Computer Science and Control (SZTAKI), Eötvös Loránd Research Network (ELKH), Machine Perception Research Laboratory, Group of Shared Virtual Reality, Budapest - Hungary
  • Arnaud Bosteels Clinique Saint-Jean, Department of Anesthesiology and Virtual Reality, Brussels - Belgium https://orcid.org/0000-0002-6818-6070




Acute care, Mixed reality, Remote assistance, Telemedicine, Ultrasound examination, Ultrasound-guided interventions


In acute care settings (emergency room [ER], intensive care unit [ICU], operating room [OR]), it is common for inexperienced physicians to have problems making an ultrasound (US) diagnosis, so they have to consult an expert. In this article, we would like to present a methodology by which geographically independent expert physicians can engage during an US examination through virtual and augmented reality. The expert can view the set-up and the US images obtained by the examiner and discuss the clinical case over video chat. In turn, s/he can advise on the proper positioning of the US transducer on the patient with the help of a virtual US transducer. This technology can be used to obtain experts’ opinion from a remote location, whether it is inside a hospital or many miles away. Similarly, it can be used for distant training; whatever the indication, it will lead to improved care. We discuss two different use cases inside an ER: US for a Focused Assessment with Sonography in Trauma (FAST) examination and US for the insertion of a central venous catheter (CVC). Currently, we position this technology to Technology Readiness Level 2, as the concept is formulated and the practical application is identified. The basic properties of algorithms are defined and the basic principles are coded. We performed experiments with parts of the technology in an artificial environment. We asked a doctor, Arnaud Bosteels, to review this method and create this article together.


Jayaprakash N, O'Sullivan R, Bey T, Ahmed SS, Lotfipour S. Crowding and delivery of healthcare in emergency departments: the European perspective. West J Emerg Med. 2009 Nov;10(4):233-239. PMID: 20046239.

State of Health in the EU, Companion Report. Imprimerie Centrale in Luxembourg; 2017. Online https://ec.europa.eu/health/system/files/2017-11/2017_companion_en_0.pdf (Accessed January 2022)

Benabbas R, Hanna M, Shah J, Sinert R. Diagnostic accuracy of history, physical examination, laboratory tests, and point-of-care ultrasound for pediatric acute appendicitis in the emergency department: a systematic review and meta-analysis. Acad Emerg Med. 2017 May;24(5):523-551. https://doi.org/10.1111/acem.13181 . PMID: 28214369. DOI: https://doi.org/10.1111/acem.13181

Javedani PP, Metzger GS, Oulton JR, Adhikari S. Use of focused assessment with sonography in trauma examination skills in the evaluation of non-trauma patients. Cureus. 2018 Jan 16;10(1):e2076. https://doi.org/10.7759/cureus.2076. PMID: 29560289. DOI: https://doi.org/10.7759/cureus.2076

Leidi A, Rouyer F, Marti C, Reny JL, Grosgurin O. Point of care ultrasonography from the emergency department to the internal medicine ward: current trends and perspectives. Intern Emerg Med. 2020;15(3):395-408. https://doi.org/10.1007/s11739-020-02284-5 PMID:32034674 DOI: https://doi.org/10.1007/s11739-020-02284-5

Micks T, Sue K, Rogers P. Barriers to point-of-care ultrasound use in rural emergency departments. Can J Emerg Med. 2016 Nov;18(6):475-479. https://doi.org/10.1017/cem.2016.337. Epub 2016 Jul 25. PMID: 27452408. DOI: https://doi.org/10.1017/cem.2016.337

Duanmu Y, Henwood PC, Takhar SS, et al. Correlation of OSCE performance and point-of-care ultrasound scan numbers among a cohort of emergency medicine residents. Ultrasound J. 2019;11(1):3. https://doi.org/10.1186/s13089-019-0118-7 PMID:31359167 DOI: https://doi.org/10.1186/s13089-019-0118-7

Costantino TG, Satz WA, Stahmer SA, Dean AJ. Predictors of success in emergency medicine ultrasound education. Acad Emerg Med. 2003 Feb;10(2):180-183. https://doi.org/10.1111/j.1553-2712.2003.tb00038.x . PMID: 12574018. DOI: https://doi.org/10.1197/aemj.10.2.180

Apertus VR. ApertusCore. [Online] SZTAKI. [Accessed January 2022] http://apertusvr.org/.

Nguyen T, Plishker W, Matisoff A, Sharma K, Shekhar R. HoloUS: augmented reality visualization of live ultrasound images using HoloLens for ultrasound-guided procedures. Int J Comput Assist Radiol Surg. 2022;17(2):385-391. https://doi.org/10.1007/s11548-021-02526-7 PMID:34817764 DOI: https://doi.org/10.1007/s11548-021-02526-7

Rüger C, Feufel MA, Moosburner S, Özbek C, Pratschke J, Sauer IM. Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions. Int J Comput Assist Radiol Surg. 2020;15(11):1895-1905. https://doi.org/10.1007/s11548-020-02236-6PMID:32725398 DOI: https://doi.org/10.1007/s11548-020-02236-6



How to Cite

Maas S, Kopacsi P, Kovacs P, Bosteels A. A mixed reality telemedicine system for collaborative ultrasound diagnostics and ultrasound-guided interventions. abtpn [Internet]. 2022 Apr. 20 [cited 2022 Sep. 26];9(1):15-20. Available from: https://journals.aboutscience.eu/index.php/aboutopen/article/view/2394



Point of view