Written by Louis Rosenberg, PhD , CEO and chief scientist of Unanimous AI:
Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where a major theme was to imagine the medical profession in the year 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical uses of virtual and augmented reality and critically assessed the current state of the field.
I have to admit, I was deeply impressed by how far augmented reality (AR) has progressed over the last eighteen months for use in medicine. So much so, that I don’t expect we’ll need to wait until 2037 for AR to have a major impact on the field. In fact, I predict that by the end of this decade augmented reality will become a common tool for surgeons, radiologists, and many other medical professionals. And by the early 2030s, many of us will go to the family doctor and be examined by a physician wearing AR glasses.
The reason is simple:
Augmented reality will give doctors superpowers.
I’m talking about superhuman capabilities for visualizing medical images, patient data, and other clinical content. The costs associated with these new capabilities are already quite reasonable and will decrease rapidly as augmented reality hardware gets produced in higher volumes in the coming years.
The first superpower is x-ray vision.
Augmented reality will give doctors the ability to peer directly into a patient and see evidence of trauma or disease at the exact location in their body where it resides. Of course, the ability to look under the skin already exists with tools like CT and MRI scanning, but currently, doctors view these images on flat screens and need to imagine how the images relate to the patient on the table. This type of mental transformation is an impressive skill, but it takes time and cognitive effort, and is not nearly as informative as it would be if doctors could simply gaze into the human body.
With AR headsets and new techniques for registering 3D medical images to a patient’s real body, the superpower of x-ray vision is now a reality. In an impressive study from Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture whole-body CT scans of trauma patients and immediately allow the medical team, all wearing AR headsets, to peer into the patient on the exam table and see the trauma in the exact location where it resides. This allowed the team to discuss the injuries and plan treatment without needing to refer back and forth to flat screens, saving time, reducing distraction, and eliminating the need for mental transformations.
In other words, AR technology takes medical images off the screen and places them in 3D space at the exact location where it’s most useful to doctors – perfectly aligned with the patient’s body. Such a capability is so natural and intuitive, that I predict it will be rapidly adopted across medical applications. In fact, I expect that in the early 2030s doctors will look back at the old way of doing things, glancing back and forth at flat screens, as awkward and primitive.
Going beyond x-ray vision, the technology of augmented reality will provide doctors with assistive content overlaid onto (and into) the patient’s body to help them with clinical tasks. For example, surgeons performing a delicate procedure will be provided with navigational cues projected on the patient in real-time, showing the exact location where interventions must be performed with precision. The objective is to increase accuracy, reduce mental effort, and speed up the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.
The concept of augmented surgery has been an aspiration of AR researchers since the core technologies were first invented. In fact, it goes back to the first AR system (the Virtual Fixtures platform) developed at Air Force Research Laboratory (AFRL) in the early 1990s. The goal of that project was to show that AR could boost human dexterity in precision tasks such as surgery. As someone who was involved in that early work, I must say that the progress the field has made over the decades since is remarkable.
Consider this – when testing that first AR system with human subjects in 1992, we required users to move metal pegs between holes spaced two feet apart in order to quantify if virtual overlays could enhance manual performance. Now, thirty years later a team at Johns Hopkins, Thomas Jefferson University Hospital, and Washington University, performed delicate spinal surgery on 28 patients using AR to assist in the placement of metal screws with precision under 2-mm. As published in a recent study, the screw-placement system achieved such accurate registration between the real patient and the virtual overlays, surgeons scored 98% on standard performance metrics.
Looking forward, we can expect augmented reality to impact all aspects of medicine as the precision has reached clinically viable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use AR in medical settings. As described above, the biggest challenge for any precision augmented reality application is accurate registration of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to accurately align the real and virtual worlds. Their method was faster and cheaper, but not quite as accurate. But this is early days – in the coming years, this technology will make AR-supported surgery viable without the need for costly markers.
This brings me to another superpower I expect doctors to have in the near future – the ability to peer back in time. That’s because physicians will be able to capture 3D images of their patients using AR headsets and later view those images aligned with their patient’s bodies. For example, a doctor could quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, interactively peering back and forth in time to compare the current view with what the lesion looked like during prior visits.
Overall, the progress being made by researchers on medical uses of virtual and augmented reality is impressive and exciting, having significant implications to both medical education and medical practice. To quote Dr. Stefano Bini of UCSF Department of Orthopaedic Surgery, “the beneficial role of AR and VR in the upskilling of the healthcare workforce cannot be overstated.”
I agree with Dr. Bini and would go even further, as I see augmented reality impacting the workforce far beyond healthcare. After all, the superpowers of x-ray vision, navigational cues, dexterity support, and the ability to peer back in time will be useful for everything from construction and auto repair to engineering, manufacture, agriculture, and of course education. And with AR glasses being developed by some of the largest companies in the world, from Microsoft and Apple, to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly come to mainstream consumers within the next five to ten years, enhancing all aspects of our daily life.
Louis Rosenberg, PhD is CEO and chief scientist of Unanimous AI and has been awarded more than 300 patents for his work in VR, AR and AI.
Shigella bacteria, which causes Shigellosis, is the primary cause of bacterial diarrhea and diarrheal death among juveniles under five years of age. Because of the antibiotic resistance of Shigella strains, no commercial vaccines are available to date. [...]
Scientists have built microscopic robots equipped with electronic “brains” that are capable of walking autonomously. A team from Cornell University in the US developed the solar-powered bots as part of research into a new generation of [...]
Blood samples from patients with long COVID who are still suffering from fatigue and shortness of breath after a year show signs of autoimmune disease, according to a study published today (Thursday) [...]
High-grade serous ovarian cancer (HGSOC) is among the deadliest human cancers and its prognosis remains extremely poor. An article published in Advanced Science explored the self-therapeutic properties of gold nanoparticles to identify a molecular axis that [...]
Antimicrobial peptides (AMPs) have a broad spectrum of antimicrobial activity and lyse microbial cells by interaction with biomembranes, offering great potential in designing new therapeutics. The antimicrobial resistance (AMR) caused due to overuse of [...]
Tumor cells are notoriously good at evading the human immune system; they put up physical walls, wear disguises and handcuff the immune system with molecular tricks. Now, UC San Francisco researchers have developed a [...]
Hyperspectral microscopy is an advanced visualization technique that combines hyperspectral imaging with state-of-the-art optics and computer software to enable rapid identification of nanomaterials. Since hyperspectral datacubes are large, their acquisition is complicated and time-consuming. [...]
Malignant brain tumors are cancerous growth in the brain with the possibility of spreading to other parts of the central nervous system (CNS). Brain tumors are highly invasive and have devastating consequences, poor prognosis, [...]
An ultrathin invention could make future computing, sensing and encryption technologies remarkably smaller and more powerful by helping scientists control a strange but useful phenomenon of quantum mechanics, according to new research recently published [...]
Methacrylate-based materials are often used in bone cement and dental resins. However, they have high failure rates as they undergo damage within ten years, impacting the quality of patient’s life and increasing healthcare costs. [...]
Early diagnosis of an infectious viral disease can help the patients and health care professionals monitor the outbreaks accurately and provide treatment at the early stage of a disease, avoiding any detrimental consequences. The [...]
Researchers at Duke University have developed a unique type of nanoparticle called a “nanorattle” that greatly enhances light emitted from within its outer shell. Loaded with light scattering dyes called Raman reporters commonly used [...]
A household microwave oven modified by a Cornell Engineering professor is helping to cook up the next generation of cellphones, computers and other electronics after the invention was shown to overcome a major challenge [...]
Nanopore sensing has emerged as a versatile approach to detecting and identifying biomolecules. Within this frame of reference, the fast-responding ionic current is considered an essential criterion for accurately measuring small objects with [...]
Graphical abstract. Credit: Journal of Photochemistry and Photobiology B: Biology (2022). DOI: 10.1016/j.jphotobiol.2022.112551 Researchers at Queen's University Belfast have developed a plastic film that can kill viruses that land on its surface with room light. The [...]
Nanotechnology that accelerates the transition of stem cells into bone could advance regenerative medicine. A nanotechnology platform developed by KAUST scientists could lead to new treatments for degenerative bone diseases. The system takes advantage [...]