One question has relentlessly followed ChatGPT in its trajectory to superstar status in the field of artificial intelligence: Has it met the Turing test of generating output indistinguishable from human response?
ChatGPT may be smart, quick and impressive. It does a good job at exhibiting apparent intelligence. It sounds humanlike in conversations with people and can even display humor, emulate the phraseology of teenagers, and pass exams for law school.
But on occasion, it has been found to serve up totally false information. It hallucinates. It does not reflect on its own output.
Cameron Jones, who specializes in language, semantics and machine learning, and Benjamin Bergen, professor of cognitive science, drew upon the work of Alan Turing, who 70 years ago devised a process to determine whether a machine could reach a point of intelligence and conversational prowess at which it could fool someone into thinking it was human.
Their report titled “Does GPT-4 Pass the Turing Test?” is available on the arXiv preprint server.
They rounded up 650 participants and generated 1,400 “games” in which brief conversations were conducted between participants and either another human or a GPT model. Participants were asked to determine who they were conversing with.
The researchers found that GPT-4 models fooled participants 41% of the time, while GPT-3.5 fooled them only 5% to 14% of the time. Interestingly, humans succeeded in convincing participants they were not machines in only 63% of the trials.
The researchers concluded, “We do not find evidence that GPT-4 passes the Turing Test.”
They noted, however, that the Turing test still retains value as a measure of the effectiveness of machine dialogue.
“The test has ongoing relevance as a framework to measure fluent social interaction and deception, and for understanding human strategies to adapt to these devices,” they said.
They warned that in many instances, chatbots can still communicate convincingly enough to fool users in many instances.
“A success rate of 41% suggests that deception by AI models may already be likely, especially in contexts where human interlocutors are less alert to the possibility they are not speaking to a human,” they said. “AI models that can robustly impersonate people could have could have widespread social and economic consequences.”
The researchers observed that participants making correct identifications focused on several factors.
Models that were too formal or too informal raised red flags for participants. If they were too wordy or too brief, if their grammar or use of punctuation was exceptionally good or “unconvincingly” bad, their usage became key factors in determining whether participants were dealing with humans or machines.
Test takers also were sensitive to generic-sounding responses.
“LLMs learn to produce highly likely completions and are fine-tuned to avoid controversial opinions. These processes might encourage generic responses that are typical overall, but lack the idiosyncrasy typical of an individual: a sort of ecological fallacy,” the researchers said.
The researchers have suggested that it will be important to track AI models as they gain more fluidity and absorb more humanlike quirks in conversation.
“It will become increasingly important to identify factors that lead to deception and strategies to mitigate it,” they said.
News
Challenging Old Theories: Innovative Microscopy Exposes New Alzheimer’s Treatment Pathways
Researchers at UC San Diego have utilized advanced imaging techniques to explore the metabolic processes behind Alzheimer’s disease, leading to potential new strategies for treatment. Alzheimer’s disease, the most common type of dementia, significantly impairs [...]
Cambridge Scientists Discover Simple “Twist” That Supercharges Clean Fuel Generation
Scientists have discovered a method to super-charge the ‘engine’ of sustainable fuel generation – by giving the materials a little twist. The researchers, led by the University of Cambridge, are developing low-cost light-harvesting semiconductors that power devices [...]
Completely New Use Discovered – This Traditional Herb Has Remarkable Nerve Regenerative Properties
Blessed thistle (Cnicus benedictus), a member of the Asteraceae family, thrives in our climate. This plant has been utilized for centuries as a medicinal herb, often consumed as an extract or tea to support [...]
Scientists study lipids cell by cell, making new cancer research possible
Imagine being able to look inside a single cancer cell and see how it communicates with its neighbors. Scientists are celebrating a new technique that lets them study the fatty contents of cancer cells, [...]
Antibiotic Breakthrough: Revolutionary Chinese Study Paves Way for Superbug Defeating Drugs
New research reveals that fluorous lipopetides act as highly effective antibiotics. Bacterial infections resistant to multiple drugs, which no existing antibiotics can treat, represent a significant worldwide challenge. A research group from China has [...]
Signs of Multiple Sclerosis Show Up in Blood Years Before Symptoms Appear
UCSF scientists clear a potential path toward earlier treatment for a disease that affects nearly 1,000,000 people in the United States. By Levi Gadye In a discovery that could hasten treatment for patients with multiple [...]
Advanced RNA Sequencing Reveals the Drivers of New COVID Variants
A study reveals that a new sequencing technique, tARC-seq, can accurately track mutations in SARS-CoV-2, providing insights into the rapid evolution and variant development of the virus. The SARS-CoV-2 virus that causes COVID has the unsettling [...]
No More Endless Boosters? Scientists Develop One-for-All Virus Vaccine
End of the line for endless boosters? Researchers at UC Riverside have developed a new vaccine approach using RNA that is effective against any strain of a virus and can be used safely even by babies or the immunocompromised. Every [...]
How Are Hydrogels Shaping the Future of Biomedicine?
Hydrogels have gained widespread recognition and utilization in biomedical engineering, with their applications dating back to the 1960s when they were first used in contact lens production. Hydrogels are distinguished from other biomaterials in [...]
Nanovials method for immune cell screening uncovers receptors that target prostate cancer
A recent UCLA study demonstrates a new process for screening T cells, part of the body's natural defenses, for characteristics vital to the success of cell-based treatments. The method filters T cells based on [...]
New Research Reveals That Your Sense of Smell May Be Smarter Than You Think
A new study published in the Journal of Neuroscience indicates that the sense of smell is significantly influenced by cues from other senses, whereas the senses of sight and hearing are much less affected. A popular [...]
Deadly bacteria show thirst for human blood: the phenomenon of bacterial vampirism
Some of the world's deadliest bacteria seek out and feed on human blood, a newly-discovered phenomenon researchers are calling "bacterial vampirism." A team led by Washington State University researchers has found the bacteria are [...]
Organ Architects: The Remarkable Cells Shaping Our Development
Finding your way through the winding streets of certain cities can be a real challenge without a map. To orient ourselves, we rely on a variety of information, including digital maps on our phones, [...]
Novel hydrogel removes microplastics from water
Microplastics pose a great threat to human health. These tiny plastic debris can enter our bodies through the water we drink and increase the risk of illnesses. They are also an environmental hazard; found [...]
Researchers Discover New Origin of Deep Brain Waves
Understanding hippocampal activity could improve sleep and cognition therapies. Researchers from the University of California, Irvine’s biomedical engineering department have discovered a new origin for two essential brain waves—slow waves and sleep spindles—that are critical for [...]
The Lifelong Cost of Surviving COVID: Scientists Uncover Long-Term Effects
Many of the individuals released to long-term acute care facilities suffered from conditions that lasted for over a year. Researchers at UC San Francisco studied COVID-19 patients in the United States who survived some of the longest and [...]