Augmented reality could soon let surgeons peer inside a patient’s body without needing to make a large incision, helping them to carry out keyhole surgery (New Scientist, 2017).

By donning an augmented reality headset, a surgeon will see a virtual 3D map of a patient’s internal organs overlaid directly onto their body as they lie on an operating table. The system has been tested using data from people projected onto a surgical mannequin during simulated operations.

Surgeons could tag virtual organs with notes before an operation to help guide their procedure, said Simon Karger, who led the development of the tech at Cambridge Consultants in Boston, Massachusetts. A doctor might highlight a nerve bundle that should be avoided or label which part of an organ should be removed, for example.

Later versions of the system could include real time feedback to help guide less experienced surgeons through complicated procedures, said Simon Karger. “We can take these very complicated specialist procedures and make them accessible to far less specialist surgeons.”

But Shafi Ahmed, a surgeon at Royal London Hospital who live streamed an operation in virtual reality last year, thinks that no amount of help from augmented reality can replace the hands on experience of a specialist surgeon. He said “We can’t trust these systems at the moment.” But he believes they could be precursors to fully automated surgical systems.

Simon Karger’s augmented reality system uses MRI and CT scans to build up a 3D image of the inside of a patient’s body, with different organs automatically colour coded by software. This image is then plugged into the Microsoft HoloLens headset, so that the surgeon sees the virtual 3D organs on the patient’s body. The system is interactive, too. Using hand gestures the surgeon can tap on an organ to remove it from their view, allowing them to take a closer look at the area they are operating on.

As the image is supposed to map exactly onto the patient, it must be pinpoint accurate to prevent surgical mistakes. Simon Karger admits it will be years before it is tested with real patients. He said “It is a very embryonic technology.”

Even so, he has a wishlist of features this kind of system could eventually have. It could be coupled with touch sensitive surgical tools that warn surgeons when they get too close to a nerve, or include virtual projections of the tools themselves so surgeons carrying out keyhole surgery can keep an eye on what they are doing without needing to look at a real time video of the operation on a screen.

Shafi Ahmed said “For certain operations and for certain specialities the system sounds very useful.”

Scopis, an augmented reality software firm, has created a similar system designed to help spinal surgeons track the position of their patient’s vertebrae during an operation. Brain surgery is another area where it might be useful to have highly detailed overlays that help a surgeon know exactly where to operate.

But Shafi Ahmed said all of these devices are still a long way from making their way into surgical theatres. They hint at what is possible but there are huge technological and regulatory hurdles to overcome before they become a reality.

Simon Karger agrees. He said “The world of surgery is quite a conservative one.” He estimates that it will only be a few years before the technology is good enough to use in real operations, but it could be decades before they are deemed safe enough for real patients.