Written by Louis Rosenberg, PhD , CEO and chief scientist of Unanimous AI:
Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where a major theme was to imagine the medical profession in the year 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical uses of virtual and augmented reality and critically assessed the current state of the field.
I have to admit, I was deeply impressed by how far augmented reality (AR) has progressed over the last eighteen months for use in medicine. So much so, that I don’t expect we’ll need to wait until 2037 for AR to have a major impact on the field. In fact, I predict that by the end of this decade augmented reality will become a common tool for surgeons, radiologists, and many other medical professionals. And by the early 2030s, many of us will go to the family doctor and be examined by a physician wearing AR glasses.
The reason is simple:
Augmented reality will give doctors superpowers.
I’m talking about superhuman capabilities for visualizing medical images, patient data, and other clinical content. The costs associated with these new capabilities are already quite reasonable and will decrease rapidly as augmented reality hardware gets produced in higher volumes in the coming years.
The first superpower is x-ray vision.
Augmented reality will give doctors the ability to peer directly into a patient and see evidence of trauma or disease at the exact location in their body where it resides. Of course, the ability to look under the skin already exists with tools like CT and MRI scanning, but currently, doctors view these images on flat screens and need to imagine how the images relate to the patient on the table. This type of mental transformation is an impressive skill, but it takes time and cognitive effort, and is not nearly as informative as it would be if doctors could simply gaze into the human body.
With AR headsets and new techniques for registering 3D medical images to a patient’s real body, the superpower of x-ray vision is now a reality. In an impressive study from Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture whole-body CT scans of trauma patients and immediately allow the medical team, all wearing AR headsets, to peer into the patient on the exam table and see the trauma in the exact location where it resides. This allowed the team to discuss the injuries and plan treatment without needing to refer back and forth to flat screens, saving time, reducing distraction, and eliminating the need for mental transformations.
In other words, AR technology takes medical images off the screen and places them in 3D space at the exact location where it’s most useful to doctors – perfectly aligned with the patient’s body. Such a capability is so natural and intuitive, that I predict it will be rapidly adopted across medical applications. In fact, I expect that in the early 2030s doctors will look back at the old way of doing things, glancing back and forth at flat screens, as awkward and primitive.
Going beyond x-ray vision, the technology of augmented reality will provide doctors with assistive content overlaid onto (and into) the patient’s body to help them with clinical tasks. For example, surgeons performing a delicate procedure will be provided with navigational cues projected on the patient in real-time, showing the exact location where interventions must be performed with precision. The objective is to increase accuracy, reduce mental effort, and speed up the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.
The concept of augmented surgery has been an aspiration of AR researchers since the core technologies were first invented. In fact, it goes back to the first AR system (the Virtual Fixtures platform) developed at Air Force Research Laboratory (AFRL) in the early 1990s. The goal of that project was to show that AR could boost human dexterity in precision tasks such as surgery. As someone who was involved in that early work, I must say that the progress the field has made over the decades since is remarkable.
Consider this – when testing that first AR system with human subjects in 1992, we required users to move metal pegs between holes spaced two feet apart in order to quantify if virtual overlays could enhance manual performance. Now, thirty years later a team at Johns Hopkins, Thomas Jefferson University Hospital, and Washington University, performed delicate spinal surgery on 28 patients using AR to assist in the placement of metal screws with precision under 2-mm. As published in a recent study, the screw-placement system achieved such accurate registration between the real patient and the virtual overlays, surgeons scored 98% on standard performance metrics.
Looking forward, we can expect augmented reality to impact all aspects of medicine as the precision has reached clinically viable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use AR in medical settings. As described above, the biggest challenge for any precision augmented reality application is accurate registration of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to accurately align the real and virtual worlds. Their method was faster and cheaper, but not quite as accurate. But this is early days – in the coming years, this technology will make AR-supported surgery viable without the need for costly markers.
This brings me to another superpower I expect doctors to have in the near future – the ability to peer back in time. That’s because physicians will be able to capture 3D images of their patients using AR headsets and later view those images aligned with their patient’s bodies. For example, a doctor could quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, interactively peering back and forth in time to compare the current view with what the lesion looked like during prior visits.
Overall, the progress being made by researchers on medical uses of virtual and augmented reality is impressive and exciting, having significant implications to both medical education and medical practice. To quote Dr. Stefano Bini of UCSF Department of Orthopaedic Surgery, “the beneficial role of AR and VR in the upskilling of the healthcare workforce cannot be overstated.”
I agree with Dr. Bini and would go even further, as I see augmented reality impacting the workforce far beyond healthcare. After all, the superpowers of x-ray vision, navigational cues, dexterity support, and the ability to peer back in time will be useful for everything from construction and auto repair to engineering, manufacture, agriculture, and of course education. And with AR glasses being developed by some of the largest companies in the world, from Microsoft and Apple, to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly come to mainstream consumers within the next five to ten years, enhancing all aspects of our daily life.
Louis Rosenberg, PhD is CEO and chief scientist of Unanimous AI and has been awarded more than 300 patents for his work in VR, AR and AI.
![](https://www.nanoappsmedical.com/wp-content/uploads/2017/05/spacer.jpg)
News
Breakthrough in Antimicrobial Technology with Cinnamon-Based Nanokiller
The need for innovative antimicrobial agents has become increasingly urgent due to the rise of antibiotic-resistant pathogens and the persistent threat of infections acquired during hospital stays. Traditional antibiotics and antiseptics are often ineffective [...]
The Silent Battle Within: How Your Organs Choose Between Mom and Dad’s Genes
Research reveals that selective expression of maternal or paternal X chromosomes varies by organ, driven by cellular competition. A new study published today (July 26) in Nature Genetics by the Lymphoid Development Group at the MRC [...]
Study identifies genes increasing risk of severe COVID-19
Whether or not a person becomes seriously ill with COVID-19 depends, among other things, on genetic factors. With this in mind, researchers from the University Hospital Bonn (UKB) and the University of Bonn, in [...]
Small regions of the brain can take micro-naps while the rest of the brain is awake and vice versa
Sleep and wake: They're totally distinct states of being that define the boundaries of our daily lives. For years, scientists have measured the difference between these instinctual brain processes by observing brain waves, with [...]
Redefining Consciousness: Small Regions of the Brain Can Take Micro-Naps While the Rest of the Brain Is Awake
The study broadly reveals how fast brain waves, previously overlooked, establish fundamental patterns of sleep and wakefulness. Scientists have developed a new method to analyze sleep and wake states by detecting ultra-fast neuronal activity [...]
AI Reveals Health Secrets Through Facial Temperature Mapping
Researchers have found that different facial temperatures correlate with chronic illnesses like diabetes and high blood pressure, and these can be detected using AI with thermal cameras. They highlight the potential of this technology [...]
Breakthrough in aging research: Blocking IL-11 extends lifespan and improves health in mice
In a recent study published in the journal Nature, a team of researchers used murine models and various pharmacological and genetic approaches to examine whether pro-inflammatory signaling involving interleukin (IL)-11, which activates signaling molecules such [...]
Promise for a universal influenza vaccine: Scientists validate theory using 1918 flu virus
New research led by Oregon Health & Science University reveals a promising approach to developing a universal influenza vaccine—a so-called "one and done" vaccine that confers lifetime immunity against an evolving virus. The study, [...]
New Projects Aim To Pioneer the Future of Neuroscience
One study will investigate the alterations in brain activity at the cellular level caused by psilocybin, the psychoactive substance found in “magic mushrooms.” How do neurons respond to the effects of magic mushrooms? What [...]
Decoding the Decline: Scientific Insights Into Long COVID’s Retreat
Research indicates a significant reduction in long COVID risk, largely due to vaccination and the virus’s evolution. The study analyzes data from over 441,000 veterans, showing lower rates of long COVID among vaccinated individuals compared [...]
Silicon Transformed: A Breakthrough in Laser Nanofabrication
A new method enables precise nanofabrication inside silicon using spatial light modulation and laser pulses, creating advanced nanostructures for potential use in electronics and photonics. Silicon, the cornerstone of modern electronics, photovoltaics, and photonics, [...]
Caught in the actinium: New research could help design better cancer treatments
The element actinium was first discovered at the turn of the 20th century, but even now, nearly 125 years later, researchers still don't have a good grasp on the metal's chemistry. That's because actinium [...]
Innovative Light-Controlled Drugs Could Revolutionize Neuropathic Pain Treatment
A team of researchers from the Institute for Bioengineering of Catalonia (IBEC) has developed light-activated derivatives of the anti-epileptic drug carbamazepine to treat neuropathic pain. Light can be harnessed to target drugs to specific [...]
Green Gold: Turning E-Waste Into a Treasure Trove of Rare Earth Metals
Scientists are developing a process inspired by nature that efficiently recovers europium from old fluorescent lamps. The approach could lead to the long-awaited recycling of rare earth metals. A small molecule that naturally serves [...]
Cambridge Study: AI Chatbots Have an “Empathy Gap,” and It Could Be Dangerous
A new study suggests a framework for “Child Safe AI” in response to recent incidents showing that many children perceive chatbots as quasi-human and reliable. A study has indicated that AI chatbots often exhibit [...]
Nanoparticle-based delivery system could offer treatment for diabetics with rare insulin allergy
Up to 3% of people with diabetes have an allergic reaction to insulin. A team at Forschungszentrum Jülich has now studied a method that could be used to deliver the active substance into the [...]