The internet search engine of the future will be powered by artificial intelligence. One can already choose from a host of AI-powered or AI-enhanced search engines—though their reliability often still leaves much to be desired. However, a team of computer scientists at the University of Massachusetts Amherst recently published and released a novel system for evaluating the reliability of AI-generated searches.
Called “eRAG,” the method is a way of putting the AI and search engine in conversation with each other, then evaluating the quality of search engines for AI use. The work is published as part of the Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.
“All of the search engines that we’ve always used were designed for humans,” says Alireza Salemi, a graduate student in the Manning College of Information and Computer Sciences at UMass Amherst and the paper’s lead author.
“They work pretty well when the user is a human, but the search engine of the future’s main user will be one of the AI Large Language Models (LLMs), like ChatGPT. This means that we need to completely redesign the way that search engines work, and my research explores how LLMs and search engines can learn from each other.”
The basic problem that Salemi and the senior author of the research, Hamed Zamani, associate professor of information and computer sciences at UMass Amherst, confront is that humans and LLMs have very different informational needs and consumption behavior.
For instance, if you can’t quite remember the title and author of that new book that was just published, you can enter a series of general search terms, such as, “what is the new spy novel with an environmental twist by that famous writer,” and then narrow the results down, or run another search as you remember more information (the author is a woman who wrote the novel “Flamethrowers”), until you find the correct result (“Creation Lake” by Rachel Kushner—which Google returned as the third hit after following the process above).
But that’s how humans work, not LLMs. They are trained on specific, enormous sets of data, and anything that is not in that data set—like the new book that just hit the stands—is effectively invisible to the LLM.
Furthermore, they’re not particularly reliable with hazy requests, because the LLM needs to be able to ask the engine for more information; but to do so, it needs to know the correct additional information to ask.
Computer scientists have devised a way to help LLMs evaluate and choose the information they need, called “retrieval-augmented generation,” or RAG. RAG is a way of augmenting LLMs with the result lists produced by search engines. But of course, the question is, how to evaluate how useful the retrieval results are for the LLMs?
So far, researchers have come up with three main ways to do this: the first is to crowdsource the accuracy of the relevance judgments with a group of humans. However, it’s a very costly method and humans may not have the same sense of relevance as an LLM.
One can also have an LLM generate a relevance judgment, which is far cheaper, but the accuracy suffers unless one has access to one of the most powerful LLM models. The third way, which is the gold standard, is to evaluate the end-to-end performance of retrieval-augmented LLMs.
But even this third method has its drawbacks. “It’s very expensive,” says Salemi, “and there are some concerning transparency issues. We don’t know how the LLM arrived at its results; we just know that it either did or didn’t.” Furthermore, there are a few dozen LLMs in existence right now, and each of them work in different ways, returning different answers.
Instead, Salemi and Zamani have developed eRAG, which is similar to the gold-standard method, but far more cost-effective, up to three times faster, uses 50 times less GPU power and is nearly as reliable.
“The first step towards developing effective search engines for AI agents is to accurately evaluate them,” says Zamani. “eRAG provides a reliable, relatively efficient and effective evaluation methodology for search engines that are being used by AI agents.”
In brief, eRAG works like this: a human user uses an LLM-powered AI agent to accomplish a task. The AI agent will submit a query to a search engine and the search engine will return a discrete number of results—say, 50—for LLM consumption.
eRAG runs each of the 50 documents through the LLM to find out which specific document the LLM found useful for generating the correct output. These document-level scores are then aggregated for evaluating the search engine quality for the AI agent.
While there is currently no search engine that can work with all the major LLMs that have been developed, the accuracy, cost-effectiveness and ease with which eRAG can be implemented is a major step toward the day when all our search engines run on AI.
This research has been awarded a Best Short Paper Award by the Association for Computing Machinery’s International Conference on Research and Development in Information Retrieval (SIGIR 2024). A public python package, containing the code for eRAG, is available at https://github.com/alirezasalemi7/eRAG.
More information: Alireza Salemi et al, Evaluating Retrieval Quality in Retrieval-Augmented Generation, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024). DOI: 10.1145/3626772.3657957
News
The Unresolved Puzzle of Long COVID: 30% of Young People Still Suffer After Two Years
A UCL study found that 70% of young people with long Covid recovered within 24 months, but recovery was less likely among older teenagers, females, and those from deprived backgrounds. Researchers emphasized the need [...]
Needle-Free: New Nano-Vaccine Effective Against All COVID-19 Variants
A new nano-vaccine developed by TAU and the University of Lisbon offers a needle-free, room-temperature-storable solution against COVID-19, targeting all key variants effectively. Professor Ronit Satchi-Fainaro’s lab at Tel Aviv University’s Faculty of Medical and [...]
Photoacoustic PDA-ICG Nanoprobe for Detecting Senescent Cells in Cancer
A study in Scientific Reports evaluated a photoacoustic polydopamine-indocyanine green (PDA-ICG) nanoprobe for detecting senescent cells. Senescent cells play a role in tumor progression and therapeutic resistance, with potential adverse effects such as inflammation and tissue [...]
How Dysregulated Cell Signaling Causes Disease
Cell signaling is crucial for cells to communicate and function correctly. Disruptions in these pathways, caused by genetic mutations or environmental factors, can lead to uncontrolled cell growth, improper immune responses, or errors in [...]
Scientists Develop Super-Strong, Eco-Friendly Plastic That Bacteria Can Eat
Researchers at the Weizmann Institute have developed a biodegradable composite material that could play a significant role in addressing the global plastic waste crisis. Billions of tons of plastic waste clutter our planet. Most [...]
Building a “Google Maps” for Biology: Human Cell Atlas Revolutionizes Medicine
New research from the Human Cell Atlas offers insights into cell development, disease mechanisms, and genetic influences, enhancing our understanding of human biology and health. The Human Cell Atlas (HCA) consortium has made significant [...]
Bioeconomic Potential: Scientists Just Found 140 Reasons to Love Spider Venom
Researchers at the LOEWE Centre for Translational Biodiversity Genomics (TBG) have discovered a significant diversity of enzymes in spider venom, previously overshadowed by the focus on neurotoxins. These enzymes, found across 140 different families, [...]
Quantum Algorithms and the Future of Precision Medicine
Precision medicine is reshaping healthcare by tailoring treatments to individual patients based on their unique genetic, environmental, and lifestyle factors. At the forefront of this revolution, the integration of quantum computing and machine learning [...]
Scientists Have Discovered a Simple Supplement That Causes Prostate Cancer Cells To Self-Destruct
Menadione, a vitamin K precursor, shows promise in slowing prostate cancer in mice by disrupting cancer cell survival processes, with potential applications for human treatment and myotubular myopathy therapy. Prostate cancer is a quiet [...]
Scientists reveal structural link for initiation of protein synthesis in bacteria
Within a cell, DNA carries the genetic code for building proteins. To build proteins, the cell makes a copy of DNA, called mRNA. Then, another molecule called a ribosome reads the mRNA, translating it [...]
Vaping Isn’t Safe: Scientists Uncover Alarming Vascular Risks
Smoking and vaping impair vascular function, even without nicotine, with the most significant effects seen in nicotine-containing e-cigarettes. Researchers recommend avoiding both for better health. Researchers have discovered immediate impacts of cigarette and e-cigarette [...]
Twice-Yearly Lenacapavir for PrEP Reduces HIV Infections by 96%
Twice-yearly injections of the capsid inhibitor drug lenacapavir can prevent the vast majority of HIV infections, according to a Phase 3 clinical trial published Wednesday in the New England Journal of Medicine. HIV pre-exposure [...]
Did Social Distancing Begin 6,000 Years Ago? Neolithic Villagers May Have Invented It
Social distancing may have roots 6,000 years ago, as research shows Neolithic villages like Nebelivka used clustered layouts to control disease spread. The phrase “social distancing” became widely recognized in recent years as people [...]
Decoding Alzheimer’s: The Arctic Mutation’s Role in Unusual Brain Structures
Researchers have uncovered how certain genetic mutations lead to unique spherical amyloid plaques in inherited forms of Alzheimer’s, offering insights that could advance our understanding of the disease and improve therapeutic strategies. An international collaboration [...]
How Your “Lizard Brain” Fuels Overthinking and Social Anxiety
New research by Northwestern Medicine reveals how humans have evolved advanced brain regions to interpret others’ thoughts, connecting these areas with the amygdala, a part of the brain involved in emotional processing. Study sought [...]
How Did Life Begin? Researchers Discover Game-Changing Clue
New research offers a potential explanation for the formation of early Earth protocells. Few questions have captivated humankind more than the mystery of life’s origins on Earth. How did the first living cells emerge? [...]