Conversational agents (CAs) like Amazon’s Alexa and Apple’s Siri are designed to answer questions, offer suggestions, and even display empathy. However, new research indicates that they fall short compared to humans in interpreting and exploring a user’s experience.
CAs are powered by large language models (LLMs) that ingest massive amounts of human-produced data, and thus can be prone to the same biases as the humans from which the information comes.
Researchers from Cornell University, Olin College, and Stanford University tested this theory by prompting CAs to display empathy while conversing with or about 65 distinct human identities.
Value Judgments and Harmful Ideologies
The team found that CAs make value judgments about certain identities – such as gay and Muslim – and can be encouraging of identities related to harmful ideologies, including Nazism.
“I think automated empathy could have tremendous impact and huge potential for positive things – for example, in education or the health care sector,” said lead author Andrea Cuadra, now a postdoctoral researcher at Stanford.
“It’s extremely unlikely that it (automated empathy) won’t happen,” she said, “so it’s important that as it’s happening, we have critical perspectives so that we can be more intentional about mitigating the potential harms.”
Cuadra will present “The Illusion of Empathy? Notes on Displays of Emotion in Human-Computer Interaction” at CHI ’24, the Association of Computing Machinery conference on Human Factors in Computing Systems, May 11-18 in Honolulu. Research co-authors at Cornell University included Nicola Dell, associate professor, Deborah Estrin, professor of computer science, and Malte Jung, associate professor of information science.
Emotional Reactions vs. Interpretations
Researchers found that, in general, LLMs received high marks for emotional reactions, but scored low for interpretations and explorations. In other words, LLMs are able to respond to a query based on their training but are unable to dig deeper.
Dell, Estrin, and Jung said they were inspired to think about this work as Cuadra was studying the use of earlier-generation CAs by older adults.
“She witnessed intriguing uses of the technology for transactional purposes such as frailty health assessments, as well as for open-ended reminiscence experiences,” Estrin said. “Along the way, she observed clear instances of the tension between compelling and disturbing ‘empathy.’”
Reference: “The Illusion of Empathy? Notes on Displays of Emotion in Human-Computer Interaction” by Andrea Cuadra, Maria Wang, Lynn Andrea Stein, Malte F. Jung, Nicola Dell, Deborah Estrin and James A. Landay, 11 May 2024, CHI ’24.
DOI: 10.1145/3613904.3642336
News
Nanomedicine in 2026: Experts Predict the Year Ahead
Progress in nanomedicine is almost as fast as the science is small. Over the last year, we've seen an abundance of headlines covering medical R&D at the nanoscale: polymer-coated nanoparticles targeting ovarian cancer, Albumin recruiting nanoparticles for [...]
Lipid nanoparticles could unlock access for millions of autoimmune patients
Capstan Therapeutics scientists demonstrate that lipid nanoparticles can engineer CAR T cells within the body without laboratory cell manufacturing and ex vivo expansion. The method using targeted lipid nanoparticles (tLNPs) is designed to deliver [...]
The Brain’s Strange Way of Computing Could Explain Consciousness
Consciousness may emerge not from code, but from the way living brains physically compute. Discussions about consciousness often stall between two deeply rooted viewpoints. One is computational functionalism, which holds that cognition can be [...]
First breathing ‘lung-on-chip’ developed using genetically identical cells
Researchers at the Francis Crick Institute and AlveoliX have developed the first human lung-on-chip model using stem cells taken from only one person. These chips simulate breathing motions and lung disease in an individual, [...]
Cell Membranes May Act Like Tiny Power Generators
Living cells may generate electricity through the natural motion of their membranes. These fast electrical signals could play a role in how cells communicate and sense their surroundings. Scientists have proposed a new theoretical [...]
This Viral RNA Structure Could Lead to a Universal Antiviral Drug
Researchers identify a shared RNA-protein interaction that could lead to broad-spectrum antiviral treatments for enteroviruses. A new study from the University of Maryland, Baltimore County (UMBC), published in Nature Communications, explains how enteroviruses begin reproducing [...]
New study suggests a way to rejuvenate the immune system
Stimulating the liver to produce some of the signals of the thymus can reverse age-related declines in T-cell populations and enhance response to vaccination. As people age, their immune system function declines. T cell [...]
Nerve Damage Can Disrupt Immunity Across the Entire Body
A single nerve injury can quietly reshape the immune system across the entire body. Preclinical research from McGill University suggests that nerve injuries may lead to long-lasting changes in the immune system, and these [...]
Fake Science Is Growing Faster Than Legitimate Research, New Study Warns
New research reveals organized networks linking paper mills, intermediaries, and compromised academic journals Organized scientific fraud is becoming increasingly common, ranging from fabricated research to the buying and selling of authorship and citations, according [...]
Scientists Unlock a New Way to Hear the Brain’s Hidden Language
Scientists can finally hear the brain’s quietest messages—unlocking the hidden code behind how neurons think, decide, and remember. Scientists have created a new protein that can capture the incoming chemical signals received by brain [...]
Does being infected or vaccinated first influence COVID-19 immunity?
A new study analyzing the immune response to COVID-19 in a Catalan cohort of health workers sheds light on an important question: does it matter whether a person was first infected or first vaccinated? [...]
We May Never Know if AI Is Conscious, Says Cambridge Philosopher
As claims about conscious AI grow louder, a Cambridge philosopher argues that we lack the evidence to know whether machines can truly be conscious, let alone morally significant. A philosopher at the University of [...]
AI Helped Scientists Stop a Virus With One Tiny Change
Using AI, researchers identified one tiny molecular interaction that viruses need to infect cells. Disrupting it stopped the virus before infection could begin. Washington State University scientists have uncovered a method to interfere with a key [...]
Deadly Hospital Fungus May Finally Have a Weakness
A deadly, drug-resistant hospital fungus may finally have a weakness—and scientists think they’ve found it. Researchers have identified a genetic process that could open the door to new treatments for a dangerous fungal infection [...]
Fever-Proof Bird Flu Variant Could Fuel the Next Pandemic
Bird flu viruses present a significant risk to humans because they can continue replicating at temperatures higher than a typical fever. Fever is one of the body’s main tools for slowing or stopping viral [...]
What could the future of nanoscience look like?
Society has a lot to thank for nanoscience. From improved health monitoring to reducing the size of electronics, scientists’ ability to delve deeper and better understand chemistry at the nanoscale has opened up numerous [...]
