How extended and virtual reality can improve medicine and surgeries

Extended reality has the potential to revolutionise the visualisation and planning of surgeries, medical experts already working with such tools told Euractiv in an interview.

Extended reality is an umbrella term that includes virtual reality (VR), augmented reality and mixed reality, referring to simulated experiences that immerse a user in a virtually-enhanced world. Among other uses such as entertainment, it can also be used to help medical professionals visualise the human body.

“In the medical field, but especially in surgery or other interventional procedures, good spatial perception is essential,” Philipp Feodorovici, Resident in Thoracic Surgery at the University Hospital in Bonn (UKB) and CTO at Bonn Surgical Technology Center (BOSTER), which is part of the University Hospital Bonn, told Euractiv.

“You need to comprehend what is located where, and so you need to have an understanding of the right three-dimensional form of a certain anatomic structure to the targeted pathology in the body,” he added.

Christof von Waldkirch, CEO of Medicalholodeck, which provides, among other things, surgical planning in augmented and virtual reality, said that “everything is 3D […] and the computer screen is flat. So, it’s just impossible to show 3D material on a computer screen. You can do it but it’s still a two-dimensional image. So, you always miss one dimension.”

“Especially in anatomy, when it comes to the human body, it just makes a difference to understand how deep the problem is. This is something that is missing in this game and spatial computing, AR, and VR solve this problem”, he added.

Feodorovici also mentioned that “there is a significant demand” for cadavers and physical models for anatomical education, in which case “you gather this three-dimensional understanding”.  However, “this is quite expensive and only available to a limited extent for the universities”.

Technology can “compensate this on a digital scale and make this kind of education available location-independent with maximum quality”, he said.

Surgical planning

“Traditionally, surgeons plan procedures based on 2D images from CT scans, MRI scans, and X-rays,” von Waldkirch said.

This is sometimes followed by creating 3D models when the case is complex and discussions with other professionals. Finally, surgeons “devise a surgical strategy based on their experience, the available 2D images, and any physical models”, he continued.

With visualisation technologies, the process still starts with “acquiring high-quality images from CT scans, MRI scans, etc”. However, this is followed by creating 3D images with which doctors can interact.

“This allows them to examine the anatomy from different angles, zoom in on areas of interest, and understand the spatial relationships between structures,” he said.

Von Waldkirch said that this could even mean possibilities for remote surgical planning by creating collaborative virtual rooms.

Robots and surgeries

Feodorovici thinks that it is important that surgeons understand the location of a tumour, for example, and how it is linked to other relevant anatomic structures because they will have to remove it without damaging anything else. Technology can help in visualising this. Moreover, many times, 2D images do not include “the human factor”.

Technology can also help “robotic-assisted surgeries” because they can be planned more precisely. Feodorovici emphasised that “it is important to understand that robotic surgery doesn’t mean that there is no human behind it”.

But they can be “highly personalised”, which is “particularly beneficial in complex cases where a one-size-fits-all approach is inadequate”, von Waldkirch said.

In the future, “the ultimate goal is to have an autonomous robotic system which doesn’t need any user input, and the surgeon will have the task of supervising the operation”, Feodorovici said.

Data collection

Both Feodorovici and von Waldkirch emphasised that most VR technologies need data acquired for treatments anyway, namely from CT and MRI. Medicalholodeck also primarily relies on this.

However, von Waldkirch explained that, in some cases, additional data may be needed, for example, biometric data, such as heart rate, blood pressure, and oxygen saturation – these could help with simulations.

“Medical data is sensitive and subject to strict privacy laws like HIPAA [Health Insurance Portability and Accountability Act] in the United States or GDPR [General Data Protection Regulation] in Europe. Programs must ensure data security and comply with these regulations,” he explained.

Moreover, patients’ data must be anonymised. Storing data can happen via clouds, which “offer accessibility but must be carefully managed for security”, von Waldkirch said.

Trust and the future

Feodorovici said it is important to think about those professionals who might not trust new technologies like virtual reality.

“There needs to be substantial research done which proves the validity of new technologies because the ultimate responsibility always lies in the hands of the attending physician,” he said.

Von Waldkirch thinks it’s important to acknowledge the limitations and that “these technologies are tools to aid, not replace, the expertise of medical professionals”. Among other aspects he mentioned the importance of safety protocols, addressing data and privacy concerns, as well as providing training.

Feodorovici also mentioned that, in the future, developing standards for the use of technology would be important.

Von Waldkirch added: “Future developments could include more detailed and realistic simulations, incorporating a wider range of physiological and anatomical details.”

[Edited by Luca Bertuzzi/Nathalie Weatherald]