Designing Intelligent Systems with Reinforcement Learning
Reinforcement learning (RL) is an AI technique that learns through trial and error, controlling an intelligent agent so that it can respond flexibly to a wide variety of situations.
For example, both videos on the right show vehicle control learned with reinforcement learning — but which one looks more intelligent? The lower one probably looks smarter, yet the upper one is actually doing something harder. The upper agent directly controls the steering angle and acceleration from its lane deviation and its distance to surrounding vehicles, whereas the lower agent does not learn vehicle control at all: from its spatial relationship to other vehicles, it learns only the high-level decision of whether to go, ease off, or stop.
So why does such a difference arise between systems that both use reinforcement learning? The key lies in system design. In this scene the objective is simply “follow the lane,” and since a well-established control law already exists, the flexibility of AI is really only needed for avoiding collisions with the vehicle ahead. Making even the vehicle control flexible, as in the upper system, therefore takes longer before the agent drives well. Of course, with more training the upper system may also achieve good control, and in more complex scenes such as intersections — where intricate decision-making and vehicle control are needed at the same time — the flexibility of reinforcement learning becomes essential for the vehicle control itself.
In our laboratory, we design “smart” intelligent systems that take these problem characteristics into account, and we aim to develop “flexible” reinforcement learning models that view the system as a whole and adapt to the diverse characteristics of a problem. We then explore how to solve real-world problems where introducing AI is difficult, by extending the capabilities of such intelligent systems through their flexibility. In particular, we study the fundamentals and applications of multi-agent reinforcement learning, targeting systems in which multiple active units — such as vehicles and robots — coexist.
Specific Research Activities in Our Laboratory
Path Planning for Multiple Robots and Smart Mobility
Safe Route Optimisation for Multiple Vessels
Our laboratory investigates route planning algorithms for multiple vessels using distributional reinforcement learning (DRL). Specifically, by combining DRL with the Obstacle Zone by Target (OZT) proposed by the National Maritime Research Institute, we have developed a method that enables vessels to learn actions that avoid collision risks based on predictions of vessel motion, thereby supporting safe navigation. Our future goals are to develop algorithms that can be applied to more advanced simulations and to construct learning algorithms that take into account real-world effects that cannot be ignored, such as wind.
A photorealistic 3D visualisation of the algorithm above (rendered in Blender; the trained policy guides every vessel to its goal, with ocean current vortices, Obstacle Zones by Target (OZT), and trajectories shown):
▶ Try steering a vessel yourself in your browser (Ship-Handling Challenge)
Route Optimisation for Multiple Aerial Delivery Robots
Japan's working-age population has continued to decline in recent years. In the transportation sector, where the impact is particularly severe, securing labour through the introduction of robots and autonomous driving technologies has become an urgent issue. In collaboration with Assistant Professor Ding at Kyoto University, Mr Aoyama at Panasonic Holdings Corporation, and Associate Professor Hayashi at Okayama University, our laboratory investigates route optimisation algorithms for aerial delivery drones. Specifically, we train drone policies in simulations that reproduce dynamic factors in aerial delivery, such as route blockages and control instability caused by wind, and develop methods that enable safe transportation even in real-world environments. Our laboratory also jointly organises the Delivery Robot Routing Problems Challenge (DRP Challenge) every year to explore better algorithms.
Multi-Objective Path Optimisation and Control for Multiple Robots
Solving real-world problems often requires handling multiple tasks, whereas AI systems are typically designed to achieve a single objective. For example, an image recognition AI aims to identify objects in an input image. However, when such recognition is applied to autonomous driving, various other objectives must also be considered, such as collision avoidance and reaching a destination. The same issue arises in multi-agent reinforcement learning.
Our laboratory addresses these limitations and investigates ways to solve real-world problems using multi-objective AI. Specifically, we improve algorithms for multi-objective multi-agent path planning and multi-objective multi-agent reinforcement learning so that they can be trained with a practical amount of learning for real-world applications.
Robot Control in Unknown Environments
Knowledge Generation and Evolution in Intelligent Robots
For autonomous mobile robots, self-localisation and path planning are essential. SLAM (Simultaneous Localisation and Mapping) is a key technology for this purpose, but discrepancies between input information and spatial relationships can cause serious problems. Our laboratory focuses on perceptual aliasing, in which different situations are incorrectly recognised as the same due to incomplete perceptual information, and works on resolving such cognitive discrepancies by drawing on insights from neuroscience.
In particular, our faculty member has worked with Professor Will N. Browne at Queensland University of Technology to improve the performance of XCS Classifier Systems, an evolutionary machine learning method, and to apply them to robot navigation.
Robots operating in real-world environments require flexible control that can adapt to uncertainty. However, AI methods, including reinforcement learning, are fundamentally designed to become intelligent through experience, which means that they often struggle with situations they have not previously encountered. Our laboratory investigates reinforcement learning methods for enabling such flexibility in field robots.
Self-Localisation Algorithms for Planetary Explorers
In the past, our faculty member investigated self-localisation algorithms for a lunar lander in the SLIM (Smart Lander for Investigating Moon) project launched by JAXA. One of the main pillars of this project was the demonstration of pinpoint landing technology based on real-time trajectory correction using image matching navigation. As the term image matching suggests, the method estimates the lander's position by matching crater patterns in lunar surface images with a pre-loaded crater map and measuring the deviation from the planned trajectory. In the self-localisation method to which our faculty member contributed, triangles are first formed from craters and matched as shown below. Then, based on the matched triangles, outlier craters are matched, and the lander's position is calculated from the coordinates of the matched craters.
Although the SLIM mission has already been completed, our laboratory continues to work on remaining challenges and aims to address them using AI-based approaches.