Written by Louis Rosenberg, PhD , CEO and chief scientist of Unanimous AI:
Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where a major theme was to imagine the medical profession in the year 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical uses of virtual and augmented reality and critically assessed the current state of the field.
I have to admit, I was deeply impressed by how far augmented reality (AR) has progressed over the last eighteen months for use in medicine. So much so, that I don’t expect we’ll need to wait until 2037 for AR to have a major impact on the field. In fact, I predict that by the end of this decade augmented reality will become a common tool for surgeons, radiologists, and many other medical professionals. And by the early 2030s, many of us will go to the family doctor and be examined by a physician wearing AR glasses.
The reason is simple:
Augmented reality will give doctors superpowers.
I’m talking about superhuman capabilities for visualizing medical images, patient data, and other clinical content. The costs associated with these new capabilities are already quite reasonable and will decrease rapidly as augmented reality hardware gets produced in higher volumes in the coming years.
The first superpower is x-ray vision.
Augmented reality will give doctors the ability to peer directly into a patient and see evidence of trauma or disease at the exact location in their body where it resides. Of course, the ability to look under the skin already exists with tools like CT and MRI scanning, but currently, doctors view these images on flat screens and need to imagine how the images relate to the patient on the table. This type of mental transformation is an impressive skill, but it takes time and cognitive effort, and is not nearly as informative as it would be if doctors could simply gaze into the human body.
With AR headsets and new techniques for registering 3D medical images to a patient’s real body, the superpower of x-ray vision is now a reality. In an impressive study from Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture whole-body CT scans of trauma patients and immediately allow the medical team, all wearing AR headsets, to peer into the patient on the exam table and see the trauma in the exact location where it resides. This allowed the team to discuss the injuries and plan treatment without needing to refer back and forth to flat screens, saving time, reducing distraction, and eliminating the need for mental transformations.
In other words, AR technology takes medical images off the screen and places them in 3D space at the exact location where it’s most useful to doctors – perfectly aligned with the patient’s body. Such a capability is so natural and intuitive, that I predict it will be rapidly adopted across medical applications. In fact, I expect that in the early 2030s doctors will look back at the old way of doing things, glancing back and forth at flat screens, as awkward and primitive.
Going beyond x-ray vision, the technology of augmented reality will provide doctors with assistive content overlaid onto (and into) the patient’s body to help them with clinical tasks. For example, surgeons performing a delicate procedure will be provided with navigational cues projected on the patient in real-time, showing the exact location where interventions must be performed with precision. The objective is to increase accuracy, reduce mental effort, and speed up the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.
The concept of augmented surgery has been an aspiration of AR researchers since the core technologies were first invented. In fact, it goes back to the first AR system (the Virtual Fixtures platform) developed at Air Force Research Laboratory (AFRL) in the early 1990s. The goal of that project was to show that AR could boost human dexterity in precision tasks such as surgery. As someone who was involved in that early work, I must say that the progress the field has made over the decades since is remarkable.
Consider this – when testing that first AR system with human subjects in 1992, we required users to move metal pegs between holes spaced two feet apart in order to quantify if virtual overlays could enhance manual performance. Now, thirty years later a team at Johns Hopkins, Thomas Jefferson University Hospital, and Washington University, performed delicate spinal surgery on 28 patients using AR to assist in the placement of metal screws with precision under 2-mm. As published in a recent study, the screw-placement system achieved such accurate registration between the real patient and the virtual overlays, surgeons scored 98% on standard performance metrics.
Looking forward, we can expect augmented reality to impact all aspects of medicine as the precision has reached clinically viable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use AR in medical settings. As described above, the biggest challenge for any precision augmented reality application is accurate registration of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to accurately align the real and virtual worlds. Their method was faster and cheaper, but not quite as accurate. But this is early days – in the coming years, this technology will make AR-supported surgery viable without the need for costly markers.
This brings me to another superpower I expect doctors to have in the near future – the ability to peer back in time. That’s because physicians will be able to capture 3D images of their patients using AR headsets and later view those images aligned with their patient’s bodies. For example, a doctor could quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, interactively peering back and forth in time to compare the current view with what the lesion looked like during prior visits.
Overall, the progress being made by researchers on medical uses of virtual and augmented reality is impressive and exciting, having significant implications to both medical education and medical practice. To quote Dr. Stefano Bini of UCSF Department of Orthopaedic Surgery, “the beneficial role of AR and VR in the upskilling of the healthcare workforce cannot be overstated.”
I agree with Dr. Bini and would go even further, as I see augmented reality impacting the workforce far beyond healthcare. After all, the superpowers of x-ray vision, navigational cues, dexterity support, and the ability to peer back in time will be useful for everything from construction and auto repair to engineering, manufacture, agriculture, and of course education. And with AR glasses being developed by some of the largest companies in the world, from Microsoft and Apple, to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly come to mainstream consumers within the next five to ten years, enhancing all aspects of our daily life.
Louis Rosenberg, PhD is CEO and chief scientist of Unanimous AI and has been awarded more than 300 patents for his work in VR, AR and AI.

News
Can our mitochondria help to beat long Covid?
At Cambridge University’s MRC Mitochondrial Biology Unit, Michal Minczuk is one of a growing number of scientists around the world aiming to find new ways of improving mitochondrial health. This line of research could help [...]
Lipid nanoparticles carry gene-editing cancer drugs past tumor defenses
As they grow, solid tumors surround themselves with a thick, hard-to-penetrate wall of molecular defenses. Getting drugs past that barricade is notoriously difficult. Now, scientists at UT Southwestern have developed nanoparticles that can break [...]
Graphene Nanosensor Detects Biomarkers Through Tears
In an article recently published in the journal Talanta, researchers demonstrated a new approach to enable the specific detection of biomarkers in human tear by employing an aptamer-based graphene affinity nanosensor. The ability to detect [...]
How Nanotechnology Can Make a Splash in Aquaculture
Selenium (Se) is an essential element found in aquatic feeds that promotes the proper development, wellbeing, and fitness of marine animals. Selenium can be transformed into nanomaterials that are more easily accessible, absorbed, and consumed by [...]
Super-Resolution Imaging Method For Multiple Fluorescence Microscopy Applications
In an article recently published in the journal Nanotechnology, researchers employed a single particle imaging method for fluorescence excitation with moderate intensity to achieve spatial resolution. Here, the semiconductor nanocrystals were accessed, whose emission lifetimes [...]
Trials to begin on new SA COVID-19 vaccine
A new COVID-19 vaccine developed in South Australia and administered with a needle-free device is to begin human trials. Designed by University of Adelaide researchers the DNA vaccine also targets the Omicron variant of [...]
Towards Carbon Clean Manufacturing with Eco-Friendly Nano-Lubricants
Grinding is an essential manufacturing process, yet the heat due to friction associated with the process causes damage to the part being processed. Lubrication is used to reduce friction; however, traditional petroleum-based lubricants can [...]
Researchers develop hybrid sensor that could help diagnose cancer
A team of researchers from HSE University, Skoltech, MPGU, and MISIS have developed a nanophotonic-microfluidic sensor whose potential applications include cancer detection, monitoring and treatment response assessment. Today, the device can identify gases and [...]
Scientists Develop ‘Nanomachines’ That Can Penetrate And Kill Cancer Cells
Researchers have made a scientific breakthrough with the development of ‘nanomachines’ that can kill cancerous cells. The research team headed by Dr Youngdo Jeong from the Center for Advanced Biomolecular Recognition at the Korea Institute of Science and Technology (KIST) has engineered [...]
Green Method to Make Nanoparticles and Ultrafine Powder
A novel freeze-dissolving approach has been devised that offers greater efficiency and sustainability compared to the classic freeze-drying process to make superfine powder or nanoparticles. In the research published in the journal ACS Sustainable Chemistry & Engineering, sphere-shaped [...]
Participants wanted for study on the regulation of what future AI-driven nanomedicines should look like
Would you like to help in some research on the regulation of what future AI-driven nanomedicines should look like? If so, researchers at the University of Bristol are looking for volunteers to discuss ethical [...]
Could gold nanoparticles help treat cancer?
Gold nanoparticles are minuscule particles made of gold. From drug and gene delivery to photothermal and photodynamic therapies to screening and diagnostic tests to radiation therapy, X-ray imaging and CT scans, these small particles [...]
Carbon Dots Target Nucleolus and Monitor in Real-Time
In an article recently published in the journal Applied Surface Science, the researchers synthesized green fluorescent carbon dots (G-CDs) from 3,5-diaminobenzoic acid and citric acid. The as-prepared G-CDs were used to target the nucleolus and [...]
Green Nanoformulation for Anti-Cancer and Antibacterial Functions
Doxorubicin (DOX) is a powerful anti-cancer medication, and efforts have been made to design nanostructures for delivering it to cancerous cells. The nanostructures increase the cytotoxic effects of DOX on cancerous cells, while reducing the negative effects [...]
New drug delivery system releases therapeutic cargo only when bacteria are present
A team of Brown University researchers has developed a new responsive material that is able to release encapsulated cargo only when pathogenic bacteria are present. The material could be used to make wound dressings [...]
Hairy Cell Leukemia Complicated by Severe COVID-19: A Case Study
Novel three-drug regimen used to manage life-threatening developments. In April 2021, a 42-year-old man reached out to Brian Hill, MD, PhD, for a second opinion after being diagnosed with hairy cell leukemia following a bone [...]