Mixed reality transforming the future of surgery

Digital & Innovation: Orthopaedic surgeon Dr Bruno Gobbato walked into an operating room in Brazil, put on a HoloLens 2 mixed-reality headset and prepared for surgery.

Joining him remotely were fellow surgeons Professor Thomas Gregory, who was tuning in from France, and Dr. John Erickson, who is based in the US. Gobbato’s patient had a collarbone fracture that had not healed sufficiently, so Gobbato needed to reposition the bone and perform a shoulder arthroscopy, which involved inserting a small camera into the joint to try to determine what was causing the man’s shoulder pain.

Gregory and Erickson were linked to Gobbato’s headset via the Microsoft Dynamics 365 Remote Assist app and shared his field of view on their computer screens through Microsoft Teams. They could see the patient and the holographic images Gobbato generated from a CT scan, one showing the patient’s damaged clavicle and another replicating his healthy clavicle. The three surgeons on three continents discussed how to approach the procedure, conferring on each step and sharing their respective approaches.

“They were my partners helping me with the surgery,” Gobbato said. “We had a French perspective, we had an American perspective and we had a Latin American perspective. We had one-quarter of the world inside the operating room.”

The procedure was part of a project aimed at showing how HoloLens 2 can benefit surgeons and enable collaboration between doctors worldwide. Conceived by Gregory, who in 2017 performed the first surgery using the original HoloLens, the initiative involved orthopaedic surgeons from 13 countries on five continents.

You may also like LifeHealthcare’s new technology to bring paradigm shift in surgical robotics

For Gregory, the project was a way to convey the benefits of what he sees as “the smartphone for surgeons.” Many people rely on smartphones to quickly access information in their daily work, he points out, but surgeons can’t use phones or computers in a sterile operating room setting. They typically plan procedures in advance and have little access to patient data or other resources once a surgery begins.

When the first HoloLens was introduced in 2016, Gregory immediately saw the potential for surgeons to use it as a tool for information and communication. Suddenly, he could use hand gestures or voice commands to call up information instead of touching a keyboard or mouse.

“It was what I was looking for for many years,” said Gregory, chief of the department of orthopedic and trauma surgery at Avicenne Hospital in Paris and a professor of orthopedic surgery at the University Sorbonne-Paris-Nord.

“I realised that I was able to use the HoloLens as a computer or a smartphone to get any information needed during surgery. That allows surgeons to be quicker, to be more efficient and to improve performance,” Gobbato said.

HoloLens is operated with hand gestures and voice commands, enabling surgeons to view three-dimensional holographic images of a patient’s anatomy created from X-rays or other scans. Surgeons can move those virtual images around to see them from different angles. They can also use the HoloLens to access patient data during surgery, call up videos or documents to help solve problems and contact other specialists for advice.

Dr. John Sledge, an orthopaedic surgeon in the US performed one of the procedures that was part of the HoloLens project, a 10-hour spinal fusion surgery. As Gregory and two surgeons from the U.K. and the UAE looked on, Sledge pulled up X-rays and scans of the patient’s lower back to locate the pieces of hardware that needed to be removed and determine how to best position his instruments to access them. He had also loaded images taken shortly after the patient’s accident so he could explain the preop history to the other surgeons.

“With the HoloLens, I can pull up the images I want and make them bigger as needed. I’ve got all the images I need literally right in front of me, in whatever size and clarity that I need them to be,” he noted.

HoloLens is also useful for planning surgeries and training. When planning a shoulder operation, for example, the surgeon can create a holographic representation of the patient’s shoulder to determine where an implant should be placed or whether the bone needs to be reconstructed.

Instead of printing out a 3D replica of a bone and having just one shot at practicing an operation on it, Sledge can test his approach on a hologram as many times as needed. For training, he can generate three-dimensional interactive models of any surgical procedure or even create simulated complications on a hologram — a fracture or unexpected bleeding — for doctors in training to solve.

“Medicine, particularly surgery, is still an apprenticeship. You watch a person operate 100 times before you’re allowed to,” Sledge said. “But now we can have residents run through 100 operations on the HoloLens, complete with rare complications and their solutions. We can do worst-case scenario training. With the HoloLens, we can make a problem occur and the doctor in training has to solve it.

“We can standardise the surgical training so that all graduates across the world will have seen and learned how to solve all of the cases in the HoloLens library in their field of training,” he said. “There’s a huge difference between you watching someone else solve a problem and you having to solve it yourself.”

Orthopaedic surgeon Dr. Stefan Greiner had never used the HoloLens before agreeing to participate in the project with Gregory. Greiner was initially sceptical about the device, seeing it as “another gadget” that might be useful in rare cases.

But as he learned how to use it, Greiner started to see more possibilities.

“I think it’s going to improve the precision of surgery and the safety of surgery, because you don’t have to look somewhere else,” noted Greiner, who runs the shoulder and elbow department at one of the biggest sports medicine facilities in Germany.

The device could also be used to help prepare patients for surgery by explaining procedures through holograms or animations.

“It’s so much more clear if you can see holograms in action, rather than just see a small picture,” he said. “This would be of great use for patients to understand what kind of surgery they are going to have.”

Dr Ashish Babhulkar had also never used a HoloLens before signing up for Gregory’s project. Babhulkar, head of the shoulder and sports injuries department at Deenanath Mangeshkar Hospital & Research Centre in Pune, India, did two reverse shoulder replacement surgeries in collaboration with Gregory and an orthopaedic surgeon from the U.K.

With input from the two doctors, Babhulkar superimposed a holographic image of the socket implant on top of the patient before inserting it to get a perfect fit. His field of view also contained a surgical manual with 3D videos for each step, which the surgeons referenced to see a device that Gregory used for shoulder surgery.

“Imagine that tomorrow in Kenya, someone’s got a HoloLens and he wants to start doing this surgery. He doesn’t have to come to India to learn it,” Babhulkar said. “We can walk him through the process without him making errors and help build his confidence. It’s going to revolutionise treatment in some remote areas.”

“The device is so powerful that as we progress on it, every year we are going to innovate and add novel applications to it,” he added.

A decade ago, it might have been hard to imagine surgeons using holographic images for surgery or envision what mixed-reality technology could enable in the medical field. The HoloLens 2 is now being used for a range of applications in healthcare, from virtual hospital rounds for medical students to treating patients with COVID-19, and a “HoloPatient” used for simulations of various medical conditions to train doctors and standardise treatment.

Elena Bonfiglioli, Microsoft’s managing director and regional lead for health care and life sciences in Europe, the Middle East and Africa, said “These leaders are reimagining surgery, which has been done in a traditional way for a very long time. The ability to imagine change and implement change is what is so transformational in these clinicians. We are really at the beginning with using HoloLens for surgery, but the path is charted and the change agents are there.

“The future of medicine is empowered by holographic computing, because holographic computing gives the new dimension of visual,” she said.

In January 2020, Microsoft partnered with Gregory and others to launch Institut Moveo, a research and teaching centre dedicated to artificial intelligence in surgery. The centre, located at University Sorbonne-Paris-Nord, brings together computer scientists, mathematicians and surgeons and provides training on using HoloLens for surgery.

Future plans include collecting HoloLens data from surgeries and using it to develop algorithms that could improve surgical procedures or even identify, based on eye movements, when a surgeon is encountering difficulty.

Gregory sees HoloLens as a device that, at a much lower cost than the surgical robots used for some types of minimally invasive surgery, which can run about $2 million, can make artificial intelligence in surgical practice accessible to even small medical facilities around the world.

“With HoloLens, we’ve got a tool to democratise this digital revolution in all of the operating theatres around the world. The impact of this is massive. I think in the future, surgeons will not be able to perform a surgery without HoloLens,” he concluded.