MIT researchers have introduced an efficient reinforcement learning algorithm that enhances AI's decision-making in complex scenarios, such as city traffic control.
By strategically selecting optimal tasks for training, the algorithm achieves significantly improved performance with far less data, offering a 50x boost in efficiency. This method not only saves time and resources but also paves the way for more effective AI applications in real-world settings.
AI Decision-Making
Across fields like robotics, medicine, and political science, researchers are working to train AI systems to make meaningful and impactful decisions. For instance, an AI system designed to manage traffic in a congested city could help drivers reach their destinations more quickly while enhancing safety and sustainability.
However, teaching AI to make effective decisions is a complex challenge.
Challenges in Reinforcement Learning
Reinforcement learning models, the foundation of many AI decision-making systems, often struggle when confronted with even slight changes in the tasks they are trained for. For example, in traffic management, a model might falter when handling intersections with varying speed limits, lane configurations, or traffic patterns.
To boost the reliability of reinforcement learning models for complex tasks with variability, MIT researchers have introduced a more efficient algorithm for training them.
Strategic Task Selection in AI Training
The algorithm strategically selects the best tasks for training an AI agent so it can effectively perform all tasks in a collection of related tasks. In the case of traffic signal control, each task could be one intersection in a task space that includes all intersections in the city.
By focusing on a smaller number of intersections that contribute the most to the algorithm's overall effectiveness, this method maximizes performance while keeping the training cost low.
Enhancing AI Efficiency With a Simple Algorithm
The researchers found that their technique was between five and 50 times more efficient than standard approaches on an array of simulated tasks. This gain in efficiency helps the algorithm learn a better solution in a faster manner, ultimately improving the performance of the AI agent.
"We were able to see incredible performance improvements, with a very simple algorithm, by thinking outside the box. An algorithm that is not very complicated stands a better chance of being adopted by the community because it is easier to implement and easier for others to understand," says senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).
She is joined on the paper by lead author Jung-Hoon Cho, a CEE graduate student; Vindula Jayawardana, a graduate student in the Department of Electrical Engineering and Computer Science (EECS); and Sirui Li, an IDSS graduate student. The research will be presented at the Conference on Neural Information Processing Systems.
Balancing Training Approaches
To train an algorithm to control traffic lights at many intersections in a city, an engineer would typically choose between two main approaches. She can train one algorithm for each intersection independently, using only that intersection's data, or train a larger algorithm using data from all intersections and then apply it to each one.
But each approach comes with its share of downsides. Training a separate algorithm for each task (such as a given intersection) is a time-consuming process that requires an enormous amount of data and computation, while training one algorithm for all tasks often leads to subpar performance.
Wu and her collaborators sought a sweet spot between these two approaches.
Advantages of Model-Based Transfer Learning
For their method, they choose a subset of tasks and train one algorithm for each task independently. Importantly, they strategically select individual tasks that are most likely to improve the algorithm's overall performance on all tasks.
They leverage a common trick from the reinforcement learning field called zero-shot transfer learning, in which an already trained model is applied to a new task without being further trained. With transfer learning, the model often performs remarkably well on the new neighbor task.
"We know it would be ideal to train on all the tasks, but we wondered if we could get away with training on a subset of those tasks, apply the result to all the tasks, and still see a performance increase," Wu says.
MBTL Algorithm: Optimizing Task Selection
To identify which tasks they should select to maximize expected performance, the researchers developed an algorithm called Model-Based Transfer Learning (MBTL).
The MBTL algorithm has two pieces. For one, it models how well each algorithm would perform if it were trained independently on one task. Then it models how much each algorithm's performance would degrade if it were transferred to each other task, a concept known as generalization performance.
Explicitly modeling generalization performance allows MBTL to estimate the value of training on a new task.
MBTL does this sequentially, choosing the task which leads to the highest performance gain first, then selecting additional tasks that provide the biggest subsequent marginal improvements to overall performance.
Since MBTL only focuses on the most promising tasks, it can dramatically improve the efficiency of the training process.
Implications for Future AI Development
When the researchers tested this technique on simulated tasks, including controlling traffic signals, managing real-time speed advisories, and executing several classic control tasks, it was five to 50 times more efficient than other methods.
This means they could arrive at the same solution by training on far less data. For instance, with a 50x efficiency boost, the MBTL algorithm could train on just two tasks and achieve the same performance as a standard method which uses data from 100 tasks.
"From the perspective of the two main approaches, that means data from the other 98 tasks was not necessary or that training on all 100 tasks is confusing to the algorithm, so the performance ends up worse than ours," Wu says.
With MBTL, adding even a small amount of additional training time could lead to much better performance.
In the future, the researchers plan to design MBTL algorithms that can extend to more complex problems, such as high-dimensional task spaces. They are also interested in applying their approach to real-world problems, especially in next-generation mobility systems.
Reference: "Model-Based Transfer Learning for Contextual Reinforcement Learning" by Jung-Hoon Cho, Vindula Jayawardana, Sirui Li and Cathy Wu, 21 November 2024, Computer Science > Machine Learning.
arXiv:2408.04498
The research is funded, in part, by a National Science Foundation CAREER Award, the Kwanjeong Educational Foundation PhD Scholarship Program, and an Amazon Robotics PhD Fellowship.
News
An Ancient Virus May Hold the Secret to What Makes the Human Brain Different
Could a virus that infected chimpanzees millions of years ago help explain how the human brain became different? Researchers at Lund University have discovered that ancient viral DNA embedded in the chimpanzee genome silences [...]
Could a Nasal Spray Stop Alzheimer’s Before Memory Loss Begins?
Protecting memory from Alzheimer’s may one day involve a nasal spray, with encouraging results in animals and evidence that its particles also protect human neurons grown in the lab. Alzheimer’s disease damages the connections [...]
A sceptical guide to AI agents in drug discovery
In conversation with Dr Eric Ma, Senior Principal Data Scientist at Moderna, Dr Raminderpal Singh examines what AI agents are actually good for, where the lock-in hides and why expertise has become more valuable [...]
Study identifies new targets for antiviral drugs against coronaviruses
A study led by researchers at the University of California San Diego School of Medicine and University of California, Riverside (UCR) School of Medicine has identified new targets for antiviral drugs against coronaviruses, including [...]
“Nearly Everybody Had Given Up” – Scientists Finally Solve a 60-Year Chemistry Mystery
After more than six decades of failed attempts, chemists have achieved a breakthrough involving a class of molecules long considered too unstable to isolate. Chemists have succeeded in isolating a group of molecules that [...]
Depression Symptoms Improved in Just 48 Hours After a Single Psychedelic Dose
A single dose of psilocybin produced rapid antidepressant effects in a small Swedish clinical trial, with improvements lasting for weeks. A single dose of psilocybin, the psychedelic compound found in magic mushrooms, produced rapid [...]
GHCE Concept
From the preface of the book Global Health Care Equivalency in the Age of Nanotechnology, Nanomedicine and Artificial Intelligence, Edited by Frank Boehm: Since the publication of my first book (Nanomedical Device and Systems [...]
Human Bodies Are Producing Two Billion Tons of CO2 a Year
A new study reveals that the amount of carbon dioxide produced by human metabolism has increased dramatically over three decades. The carbon dioxide produced by human bodies as they convert food into energy increased [...]
A Hidden DNA Mechanism Could Explain How 20% of Human Cancers Develop
Researchers have identified an unexpected molecular vulnerability in cancers carrying BAF mutations. Mutations found in roughly 20% of human cancers may promote tumor growth by changing how DNA is organized inside cells. Researchers have [...]
Ebola kills over 5,300 in fastest-growing outbreak on record
Africa’s latest Ebola outbreak has killed nearly 5,400 people, the vast majority of them in the Democratic Republic of Congo (DRC), according to the continent’s top health agency. Africa Centres for Disease Control and [...]
China makes early progress in using sound waves to stimulate damaged brains
Chinese researchers are testing whether focused ultrasound can help restore consciousness in patients with severe brain injuries, adding to a growing international effort to use sound waves to stimulate deep brain structures without surgery. [...]
Micro-drones fly through the body thanks to ultrasound
At the EPFL, micro-drones have been developed that can be moved without a motor or battery.No motors, no batteries, no electronics on board, just small cavities that convert sound waves into thrust: this is [...]
A Single Enzyme Could Hold the Key to Treating a Devastating Brain Disorder
Blocking an enzyme’s production restored brain growth and healthy behaviors in mice engineered to develop Angelman syndrome, pointing to a possible treatment target. Children with Angelman syndrome begin to show difficulties with thinking and [...]
Scientists Develop a Clever New Way To Kill Cancer Cells From the Inside
Scientists have developed an experimental strategy that exploits a chemical byproduct of cancer cells to produce anticancer drugs directly inside them. Scientists in Japan have developed an experimental method that uses a chemical produced [...]
Our books are available worldwide!
Online Sellers other than Amazon, Routledge, and IOPP Indigo Global Health Care Equivalency in the Age of Nanotechnology, Nanomedicine and Artifcial Intelligence Global Health Care Equivalency In The Age Of Nanotechnology, Nanomedicine And Artificial [...]
After only a short training, people in a coma managed to send targeted signals
Researchers have succeeded in communicating with brain-injured patients who are unconscious. They see enormous potential in this for future treatments. Patients who lie motionless in bed after a severe brain injury and do not [...]















