One question has relentlessly followed ChatGPT in its trajectory to superstar status in the field of artificial intelligence: Has it met the Turing test of generating output indistinguishable from human response?
ChatGPT may be smart, quick and impressive. It does a good job at exhibiting apparent intelligence. It sounds humanlike in conversations with people and can even display humor, emulate the phraseology of teenagers, and pass exams for law school.
But on occasion, it has been found to serve up totally false information. It hallucinates. It does not reflect on its own output.
Cameron Jones, who specializes in language, semantics and machine learning, and Benjamin Bergen, professor of cognitive science, drew upon the work of Alan Turing, who 70 years ago devised a process to determine whether a machine could reach a point of intelligence and conversational prowess at which it could fool someone into thinking it was human.
Their report titled “Does GPT-4 Pass the Turing Test?” is available on the arXiv preprint server.
They rounded up 650 participants and generated 1,400 “games” in which brief conversations were conducted between participants and either another human or a GPT model. Participants were asked to determine who they were conversing with.
The researchers found that GPT-4 models fooled participants 41% of the time, while GPT-3.5 fooled them only 5% to 14% of the time. Interestingly, humans succeeded in convincing participants they were not machines in only 63% of the trials.
The researchers concluded, “We do not find evidence that GPT-4 passes the Turing Test.”
They noted, however, that the Turing test still retains value as a measure of the effectiveness of machine dialogue.
“The test has ongoing relevance as a framework to measure fluent social interaction and deception, and for understanding human strategies to adapt to these devices,” they said.
They warned that in many instances, chatbots can still communicate convincingly enough to fool users in many instances.
“A success rate of 41% suggests that deception by AI models may already be likely, especially in contexts where human interlocutors are less alert to the possibility they are not speaking to a human,” they said. “AI models that can robustly impersonate people could have could have widespread social and economic consequences.”
The researchers observed that participants making correct identifications focused on several factors.
Models that were too formal or too informal raised red flags for participants. If they were too wordy or too brief, if their grammar or use of punctuation was exceptionally good or “unconvincingly” bad, their usage became key factors in determining whether participants were dealing with humans or machines.
Test takers also were sensitive to generic-sounding responses.
“LLMs learn to produce highly likely completions and are fine-tuned to avoid controversial opinions. These processes might encourage generic responses that are typical overall, but lack the idiosyncrasy typical of an individual: a sort of ecological fallacy,” the researchers said.
The researchers have suggested that it will be important to track AI models as they gain more fluidity and absorb more humanlike quirks in conversation.
“It will become increasingly important to identify factors that lead to deception and strategies to mitigate it,” they said.
News
Nerve Damage Can Disrupt Immunity Across the Entire Body
A single nerve injury can quietly reshape the immune system across the entire body. Preclinical research from McGill University suggests that nerve injuries may lead to long-lasting changes in the immune system, and these [...]
Fake Science Is Growing Faster Than Legitimate Research, New Study Warns
New research reveals organized networks linking paper mills, intermediaries, and compromised academic journals Organized scientific fraud is becoming increasingly common, ranging from fabricated research to the buying and selling of authorship and citations, according [...]
Scientists Unlock a New Way to Hear the Brain’s Hidden Language
Scientists can finally hear the brain’s quietest messages—unlocking the hidden code behind how neurons think, decide, and remember. Scientists have created a new protein that can capture the incoming chemical signals received by brain [...]
Does being infected or vaccinated first influence COVID-19 immunity?
A new study analyzing the immune response to COVID-19 in a Catalan cohort of health workers sheds light on an important question: does it matter whether a person was first infected or first vaccinated? [...]
We May Never Know if AI Is Conscious, Says Cambridge Philosopher
As claims about conscious AI grow louder, a Cambridge philosopher argues that we lack the evidence to know whether machines can truly be conscious, let alone morally significant. A philosopher at the University of [...]
AI Helped Scientists Stop a Virus With One Tiny Change
Using AI, researchers identified one tiny molecular interaction that viruses need to infect cells. Disrupting it stopped the virus before infection could begin. Washington State University scientists have uncovered a method to interfere with a key [...]
Deadly Hospital Fungus May Finally Have a Weakness
A deadly, drug-resistant hospital fungus may finally have a weakness—and scientists think they’ve found it. Researchers have identified a genetic process that could open the door to new treatments for a dangerous fungal infection [...]
Fever-Proof Bird Flu Variant Could Fuel the Next Pandemic
Bird flu viruses present a significant risk to humans because they can continue replicating at temperatures higher than a typical fever. Fever is one of the body’s main tools for slowing or stopping viral [...]
What could the future of nanoscience look like?
Society has a lot to thank for nanoscience. From improved health monitoring to reducing the size of electronics, scientists’ ability to delve deeper and better understand chemistry at the nanoscale has opened up numerous [...]
Scientists Melt Cancer’s Hidden “Power Hubs” and Stop Tumor Growth
Researchers discovered that in a rare kidney cancer, RNA builds droplet-like hubs that act as growth control centers inside tumor cells. By engineering a molecular switch to dissolve these hubs, they were able to halt cancer [...]
Platelet-inspired nanoparticles could improve treatment of inflammatory diseases
Scientists have developed platelet-inspired nanoparticles that deliver anti-inflammatory drugs directly to brain-computer interface implants, doubling their effectiveness. Scientists have found a way to improve the performance of brain-computer interface (BCI) electrodes by delivering anti-inflammatory drugs directly [...]
After 150 years, a new chapter in cancer therapy is finally beginning
For decades, researchers have been looking for ways to destroy cancer cells in a targeted manner without further weakening the body. But for many patients whose immune system is severely impaired by chemotherapy or radiation, [...]
Older chemical libraries show promise for fighting resistant strains of COVID-19 virus
SARS‑CoV‑2, the virus that causes COVID-19, continues to mutate, with some newer strains becoming less responsive to current antiviral treatments like Paxlovid. Now, University of California San Diego scientists and an international team of [...]
Lower doses of immunotherapy for skin cancer give better results, study suggests
According to a new study, lower doses of approved immunotherapy for malignant melanoma can give better results against tumors, while reducing side effects. This is reported by researchers at Karolinska Institutet in the Journal of the National [...]
Researchers highlight five pathways through which microplastics can harm the brain
Microplastics could be fueling neurodegenerative diseases like Alzheimer's and Parkinson's, with a new study highlighting five ways microplastics can trigger inflammation and damage in the brain. More than 57 million people live with dementia, [...]
Tiny Metal Nanodots Obliterate Cancer Cells While Largely Sparing Healthy Tissue
Scientists have developed tiny metal-oxide particles that push cancer cells past their stress limits while sparing healthy tissue. An international team led by RMIT University has developed tiny particles called nanodots, crafted from a metallic compound, [...]
