Written by Louis Rosenberg, PhD , CEO and chief scientist of Unanimous AI:
Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where a major theme was to imagine the medical profession in the year 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical uses of virtual and augmented reality and critically assessed the current state of the field.
I have to admit, I was deeply impressed by how far augmented reality (AR) has progressed over the last eighteen months for use in medicine. So much so, that I don’t expect we’ll need to wait until 2037 for AR to have a major impact on the field. In fact, I predict that by the end of this decade augmented reality will become a common tool for surgeons, radiologists, and many other medical professionals. And by the early 2030s, many of us will go to the family doctor and be examined by a physician wearing AR glasses.
The reason is simple:
Augmented reality will give doctors superpowers.
I’m talking about superhuman capabilities for visualizing medical images, patient data, and other clinical content. The costs associated with these new capabilities are already quite reasonable and will decrease rapidly as augmented reality hardware gets produced in higher volumes in the coming years.
The first superpower is x-ray vision.
Augmented reality will give doctors the ability to peer directly into a patient and see evidence of trauma or disease at the exact location in their body where it resides. Of course, the ability to look under the skin already exists with tools like CT and MRI scanning, but currently, doctors view these images on flat screens and need to imagine how the images relate to the patient on the table. This type of mental transformation is an impressive skill, but it takes time and cognitive effort, and is not nearly as informative as it would be if doctors could simply gaze into the human body.
With AR headsets and new techniques for registering 3D medical images to a patient’s real body, the superpower of x-ray vision is now a reality. In an impressive study from Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture whole-body CT scans of trauma patients and immediately allow the medical team, all wearing AR headsets, to peer into the patient on the exam table and see the trauma in the exact location where it resides. This allowed the team to discuss the injuries and plan treatment without needing to refer back and forth to flat screens, saving time, reducing distraction, and eliminating the need for mental transformations.
In other words, AR technology takes medical images off the screen and places them in 3D space at the exact location where it’s most useful to doctors – perfectly aligned with the patient’s body. Such a capability is so natural and intuitive, that I predict it will be rapidly adopted across medical applications. In fact, I expect that in the early 2030s doctors will look back at the old way of doing things, glancing back and forth at flat screens, as awkward and primitive.
Going beyond x-ray vision, the technology of augmented reality will provide doctors with assistive content overlaid onto (and into) the patient’s body to help them with clinical tasks. For example, surgeons performing a delicate procedure will be provided with navigational cues projected on the patient in real-time, showing the exact location where interventions must be performed with precision. The objective is to increase accuracy, reduce mental effort, and speed up the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.
The concept of augmented surgery has been an aspiration of AR researchers since the core technologies were first invented. In fact, it goes back to the first AR system (the Virtual Fixtures platform) developed at Air Force Research Laboratory (AFRL) in the early 1990s. The goal of that project was to show that AR could boost human dexterity in precision tasks such as surgery. As someone who was involved in that early work, I must say that the progress the field has made over the decades since is remarkable.
Consider this – when testing that first AR system with human subjects in 1992, we required users to move metal pegs between holes spaced two feet apart in order to quantify if virtual overlays could enhance manual performance. Now, thirty years later a team at Johns Hopkins, Thomas Jefferson University Hospital, and Washington University, performed delicate spinal surgery on 28 patients using AR to assist in the placement of metal screws with precision under 2-mm. As published in a recent study, the screw-placement system achieved such accurate registration between the real patient and the virtual overlays, surgeons scored 98% on standard performance metrics.
Looking forward, we can expect augmented reality to impact all aspects of medicine as the precision has reached clinically viable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use AR in medical settings. As described above, the biggest challenge for any precision augmented reality application is accurate registration of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to accurately align the real and virtual worlds. Their method was faster and cheaper, but not quite as accurate. But this is early days – in the coming years, this technology will make AR-supported surgery viable without the need for costly markers.
This brings me to another superpower I expect doctors to have in the near future – the ability to peer back in time. That’s because physicians will be able to capture 3D images of their patients using AR headsets and later view those images aligned with their patient’s bodies. For example, a doctor could quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, interactively peering back and forth in time to compare the current view with what the lesion looked like during prior visits.
Overall, the progress being made by researchers on medical uses of virtual and augmented reality is impressive and exciting, having significant implications to both medical education and medical practice. To quote Dr. Stefano Bini of UCSF Department of Orthopaedic Surgery, “the beneficial role of AR and VR in the upskilling of the healthcare workforce cannot be overstated.”
I agree with Dr. Bini and would go even further, as I see augmented reality impacting the workforce far beyond healthcare. After all, the superpowers of x-ray vision, navigational cues, dexterity support, and the ability to peer back in time will be useful for everything from construction and auto repair to engineering, manufacture, agriculture, and of course education. And with AR glasses being developed by some of the largest companies in the world, from Microsoft and Apple, to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly come to mainstream consumers within the next five to ten years, enhancing all aspects of our daily life.
Louis Rosenberg, PhD is CEO and chief scientist of Unanimous AI and has been awarded more than 300 patents for his work in VR, AR and AI.

News
A Promising New Pathway in the Battle Against Aggressive Prostate Cancer
Neuronal Molecule Makes Prostate Cancer More Aggressive Researchers discover a potential therapeutic avenue against an aggressive form of prostate cancer. Prostate cancer is the second most common cancer and the second leading cause of [...]
Nasal Vaccines: Stopping the COVID-19 Virus Before It Reaches the Lungs
The Pfizer-BioNTech and Moderna mRNA vaccines have played a large role in preventing deaths and severe infections from COVID-19. But researchers are still in the process of developing alternative approaches to vaccines to improve [...]
NASA Tracking a Huge, Growing Anomaly in Earth’s Magnetic Field – with video
NASA is actively monitoring a strange anomaly in Earth's magnetic field: a giant region of lower magnetic intensity in the skies above the planet, stretching out between South America and southwest Africa. This vast, developing [...]
New, Better Models Show How Infectious Diseases Like COVID-19 Spread
Infectious diseases such as COVID-19 can spread rapidly across the globe. Models that can predict how such diseases spread will strengthen national surveillance systems and improve public health decision-making. The COVID-19 pandemic has emphasized the [...]
Human Antibodies Discovered That Can Block Multiple Coronaviruses Including COVID-19
Results from a Scripps Research and UNC team pave the way for a vaccine and therapeutic antibodies that could be stockpiled to fight future coronavirus pandemics. A team of scientists from Scripps Research and [...]
Nanotechnology could be used to treat lymphedema
The human body is made up of thousands of tiny lymphatic vessels that ferry white blood cells and proteins around the body, like a superhighway of the immune system. It's remarkably efficient, but if [...]
DNA Nanotechnology Tools – From Design to Applications
Suite of DNA nanotechnology devices engineered to overcome specific bottlenecks in the development of new therapies, diagnostics, and understanding of molecular structures. DNA nanostructures with their potential for cell and tissue permeability, biocompatibility, and [...]
Regenerating bone with deer antler stem cells
Scientists from a collection of Chinese research institutions collaborated on a study of organ regeneration in mammals, finding deer antler blastema progenitor cells are a possible source of conserved regeneration cells in higher vertebrates. [...]
AI Takes On Cancer: Analysis of Mutations Could Lead to Improved Therapy
Cancer is a complex and diverse disease, and its range of associated mutations is vast. The combination of these genomic changes in an individual is referred to as their “mutational landscape.” These landscapes vary [...]
Exposing tumours to bacteria converts immune cells to cancer killers
New research on inflammation could lead to better treatments to improve outcomes for people with advanced or previously untreatable cancers. Introducing bacteria to a tumour’s microenvironment creates a state of acute inflammation that triggers [...]
Smart nanotechnology for more accurate delivery of insulin
More efficient and longer lasting glucose-responsive insulin that eliminates the need for people with type 1 diabetes to measure their glucose levels could be a step closer thanks to a Monash University-led project. Published [...]
Efficiently Harvesting Rare Earth Elements From Wastewater Using Exotic Bacteria
The novel strains of cyanobacteria exhibit a fast and efficient “biosorption” of rare earth elements, making recycling possible. Rare earth elements (REEs) are a set of 17 metallic elements that possess similar chemical properties. [...]
Resisting Treatment: Cancer Cells Shrink or Super-Size To Survive
A new approach to image analysis has uncovered how cancer cells manipulate their size as a means of resisting treatment. Researchers have discovered that cancer cells are capable of either shrinking or super-size themselves [...]
New Research Explains Why Children Avoid Severe COVID-19 Symptoms
According to new research, children exhibit a robust initial immune response to the coronavirus, however, they are unable to transfer this response to long-lasting memory T cells like adults do. Researchers led by scientists [...]
Scientists Unravel Protein Map of Mitochondria
A new study sheds light on the organization of proteins within mitochondria. Mitochondria, the “powerhouses” of cells, play a crucial role in the energy production of organisms and are involved in various metabolic and [...]
Mystifying Trapping Phenomenon: A Surprising Way To Catch a Microparticle
New insights could advance microfluidics and drug delivery systems. New study finds obstacles can trap rolling microparticles in fluid Through simulations and experiments, physicists attribute the trapping effect to stagnant pockets of fluid, created [...]