The internet search engine of the future will be powered by artificial intelligence. One can already choose from a host of AI-powered or AI-enhanced search engines—though their reliability often still leaves much to be desired. However, a team of computer scientists at the University of Massachusetts Amherst recently published and released a novel system for evaluating the reliability of AI-generated searches.
Called “eRAG,” the method is a way of putting the AI and search engine in conversation with each other, then evaluating the quality of search engines for AI use. The work is published as part of the Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.
“All of the search engines that we’ve always used were designed for humans,” says Alireza Salemi, a graduate student in the Manning College of Information and Computer Sciences at UMass Amherst and the paper’s lead author.
“They work pretty well when the user is a human, but the search engine of the future’s main user will be one of the AI Large Language Models (LLMs), like ChatGPT. This means that we need to completely redesign the way that search engines work, and my research explores how LLMs and search engines can learn from each other.”
The basic problem that Salemi and the senior author of the research, Hamed Zamani, associate professor of information and computer sciences at UMass Amherst, confront is that humans and LLMs have very different informational needs and consumption behavior.
For instance, if you can’t quite remember the title and author of that new book that was just published, you can enter a series of general search terms, such as, “what is the new spy novel with an environmental twist by that famous writer,” and then narrow the results down, or run another search as you remember more information (the author is a woman who wrote the novel “Flamethrowers”), until you find the correct result (“Creation Lake” by Rachel Kushner—which Google returned as the third hit after following the process above).
But that’s how humans work, not LLMs. They are trained on specific, enormous sets of data, and anything that is not in that data set—like the new book that just hit the stands—is effectively invisible to the LLM.
Furthermore, they’re not particularly reliable with hazy requests, because the LLM needs to be able to ask the engine for more information; but to do so, it needs to know the correct additional information to ask.
Computer scientists have devised a way to help LLMs evaluate and choose the information they need, called “retrieval-augmented generation,” or RAG. RAG is a way of augmenting LLMs with the result lists produced by search engines. But of course, the question is, how to evaluate how useful the retrieval results are for the LLMs?
So far, researchers have come up with three main ways to do this: the first is to crowdsource the accuracy of the relevance judgments with a group of humans. However, it’s a very costly method and humans may not have the same sense of relevance as an LLM.
One can also have an LLM generate a relevance judgment, which is far cheaper, but the accuracy suffers unless one has access to one of the most powerful LLM models. The third way, which is the gold standard, is to evaluate the end-to-end performance of retrieval-augmented LLMs.
But even this third method has its drawbacks. “It’s very expensive,” says Salemi, “and there are some concerning transparency issues. We don’t know how the LLM arrived at its results; we just know that it either did or didn’t.” Furthermore, there are a few dozen LLMs in existence right now, and each of them work in different ways, returning different answers.
Instead, Salemi and Zamani have developed eRAG, which is similar to the gold-standard method, but far more cost-effective, up to three times faster, uses 50 times less GPU power and is nearly as reliable.
“The first step towards developing effective search engines for AI agents is to accurately evaluate them,” says Zamani. “eRAG provides a reliable, relatively efficient and effective evaluation methodology for search engines that are being used by AI agents.”
In brief, eRAG works like this: a human user uses an LLM-powered AI agent to accomplish a task. The AI agent will submit a query to a search engine and the search engine will return a discrete number of results—say, 50—for LLM consumption.
eRAG runs each of the 50 documents through the LLM to find out which specific document the LLM found useful for generating the correct output. These document-level scores are then aggregated for evaluating the search engine quality for the AI agent.
While there is currently no search engine that can work with all the major LLMs that have been developed, the accuracy, cost-effectiveness and ease with which eRAG can be implemented is a major step toward the day when all our search engines run on AI.
This research has been awarded a Best Short Paper Award by the Association for Computing Machinery’s International Conference on Research and Development in Information Retrieval (SIGIR 2024). A public python package, containing the code for eRAG, is available at https://github.com/alirezasalemi7/eRAG.
More information: Alireza Salemi et al, Evaluating Retrieval Quality in Retrieval-Augmented Generation, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024). DOI: 10.1145/3626772.3657957

News
Repurposed drugs could calm the immune system’s response to nanomedicine
An international study led by researchers at the University of Colorado Anschutz Medical Campus has identified a promising strategy to enhance the safety of nanomedicines, advanced therapies often used in cancer and vaccine treatments, [...]
Nano-Enhanced Hydrogel Strategies for Cartilage Repair
A recent article in Engineering describes the development of a protein-based nanocomposite hydrogel designed to deliver two therapeutic agents—dexamethasone (Dex) and kartogenin (KGN)—to support cartilage repair. The hydrogel is engineered to modulate immune responses and promote [...]
New Cancer Drug Blocks Tumors Without Debilitating Side Effects
A new drug targets RAS-PI3Kα pathways without harmful side effects. It was developed using high-performance computing and AI. A new cancer drug candidate, developed through a collaboration between Lawrence Livermore National Laboratory (LLNL), BridgeBio Oncology [...]
Scientists Are Pretty Close to Replicating the First Thing That Ever Lived
For 400 million years, a leading hypothesis claims, Earth was an “RNA World,” meaning that life must’ve first replicated from RNA before the arrival of proteins and DNA. Unfortunately, scientists have failed to find [...]
Why ‘Peniaphobia’ Is Exploding Among Young People (And Why We Should Be Concerned)
An insidious illness is taking hold among a growing proportion of young people. Little known to the general public, peniaphobia—the fear of becoming poor—is gaining ground among teens and young adults. Discover the causes [...]
Team finds flawed data in recent study relevant to coronavirus antiviral development
The COVID pandemic illustrated how urgently we need antiviral medications capable of treating coronavirus infections. To aid this effort, researchers quickly homed in on part of SARS-CoV-2's molecular structure known as the NiRAN domain—an [...]
Drug-Coated Neural Implants Reduce Immune Rejection
Summary: A new study shows that coating neural prosthetic implants with the anti-inflammatory drug dexamethasone helps reduce the body’s immune response and scar tissue formation. This strategy enhances the long-term performance and stability of electrodes [...]
Scientists discover cancer-fighting bacteria that ‘soak up’ forever chemicals in the body
A family of healthy bacteria may help 'soak up' toxic forever chemicals in the body, warding off their cancerous effects. Forever chemicals, also known as PFAS (per- and polyfluoroalkyl substances), are toxic chemicals that [...]
Johns Hopkins Researchers Uncover a New Way To Kill Cancer Cells
A new study reveals that blocking ribosomal RNA production rewires cancer cell behavior and could help treat genetically unstable tumors. Researchers at the Johns Hopkins Kimmel Cancer Center and the Department of Radiation Oncology and Molecular [...]
AI matches doctors in mapping lung tumors for radiation therapy
In radiation therapy, precision can save lives. Oncologists must carefully map the size and location of a tumor before delivering high-dose radiation to destroy cancer cells while sparing healthy tissue. But this process, called [...]
Scientists Finally “See” Key Protein That Controls Inflammation
Researchers used advanced microscopy to uncover important protein structures. For the first time, two important protein structures in the human body are being visualized, thanks in part to cutting-edge technology at the University of [...]
AI tool detects 9 types of dementia from a single brain scan
Mayo Clinic researchers have developed a new artificial intelligence (AI) tool that helps clinicians identify brain activity patterns linked to nine types of dementia, including Alzheimer's disease, using a single, widely available scan—a transformative [...]
Is plastic packaging putting more than just food on your plate?
New research reveals that common food packaging and utensils can shed microscopic plastics into our food, prompting urgent calls for stricter testing and updated regulations to protect public health. Beyond microplastics: The analysis intentionally [...]
Aging Spreads Through the Bloodstream
Summary: New research reveals that aging isn’t just a local cellular process—it can spread throughout the body via the bloodstream. A redox-sensitive protein called ReHMGB1, secreted by senescent cells, was found to trigger aging features [...]
AI and nanomedicine find rare biomarkers for prostrate cancer and atherosclerosis
Imagine a stadium packed with 75,000 fans, all wearing green and white jerseys—except one person in a solid green shirt. Finding that person would be tough. That's how hard it is for scientists to [...]
Are Pesticides Breeding the Next Pandemic? Experts Warn of Fungal Superbugs
Fungicides used in agriculture have been linked to an increase in resistance to antifungal drugs in both humans and animals. Fungal infections are on the rise, and two UC Davis infectious disease experts, Dr. George Thompson [...]