I Introduction
Neurosymbolic Artificial Intelligence (Neurosymbolic AI), a budding field of Artificial Intelligence(AI), has garnered significant attention in recent times as it combines both neural and symbolic traditions to enhance the performance of neural network models. In this context, the term ”neural” pertains to neural network primarily, while ”symbolic” refers to the use of various mathematical logic and algorithms for symbolic manipulation. Reinforcement Learning (RL), another emerging area of machine learning, revolves around agents operating in various environments to maximize their rewards. It dates back to the early days of cybernetics and has gained rapid interest in the machine learning and AI communities over the last five to ten years. RL involves programming agents by reward and punishment without specifying how to accomplish the task, and it encompasses statistics, psychology, neuroscience, and computer science. However, there are significant computational challenges to overcome[1]. Deep Reinforcement Learning (DRL), which replaces tabular methods of estimating state values with function approximation, has eliminated the need to store all state value pairs in the table, enabling the agent to generalize the value of states that it has never encountered before. DRL has been utilized in programs that have defeated the best human players in game of Go[2]. Additionally, an AI agent named AlphaStar[3] beat the world’s best StarCraft II player.
RL has recently drawn much attention in the context of Neurosymbolic AI for policy synthesis and representation. Neurosymbolic Reinforcement Learning(Neurosymbolic RL) merge planning-style control-flow instructions with fundamental atomic actions that are learned and represented through Deep Neural Networks(DNNs). The combination of neural and symbolic approaches enables the efficient use of DRL techniques to improve the interpretability and transparency of an agent’s behavior while also leveraging a high-level, symbolic representation of the policies learned by agents. By allowing the neural system to interact with the knowledge base, the reasoning ability is enhanced, and the learning ability is enhanced by interacting with the neural system. This interaction results in better generalization and transfer of knowledge, improved efficiency and robustness, and an increase in explanation and interpretability. Fig.1 illustrates the general idea of combining Neurosymbolic AI with RL to give rise to Neurosymbolic RL. Neurosymbolic agent has advantage of using both neural and symbolic counterpart so as to enhance not only its learning ability but also reasoning skills. Further advantage of such kind of model can be in the areas like reward shaping, reducing the complexity of the environment and also synthesizing the efficient symbolic policy. However this approach is on its primitive phase much of the areas are still worth exploration.
Further research is needed in Neurosymbolic RL to develop novel approaches, techniques, and their real-time applications that best fit real-world use cases such as computer networks, healthcare, IoT devices, finance, and other industrial domains. In this paper, we have analyzed notable works in Neurosymbolic RL to date. We have examined the neural and symbolic component used in each of the research work. Further we analyzed RL components used in the architecture: RL-algorithms, state space, action space, and policy module used so that we can have transparent view of the working of the model. We have classified these research works into three main categories: Learning for Reasoning RL model, Reasoning for Learning RL model, and Learning-Reasoning RL model, which are further sub-divided according to the significance or role of the model. After that we move on to provide a comprehensive summary of the Neurosymbolic RL approaches related to their specific application cases that need to be developed to meet the needs of AI. Finally, we have presented certain challenges specific to each application case to employ them in real-world scenarios.
In reviewing the history of AI surveys, no significant work has been found that specifically focused on the combination of Neurosymbolic AI with RL. Earlier works are either focused solely on Neurosymbolic AI or on the field of RL. Negligible surveys can be found on Neurosymbolic AI[4, 5] few other provides insight on recent advances[6] and application[7]. A large number of surveys are available on RL on various aspects:
- RL in general[1], DRL[8, 9], Causal RL[10]
- Safety and Security in RL [11, 12]
- Environment [13]
- Agent [14, 15, 16]
- Application like Natural Language Processing [17, 18], Communication Network [19], Robotics [20], Healthcare [21], Transportation [22]
This survey is the first of its kind and the first attempt to evaluate the combination of these two popular areas as one (Neurosymbolic RL). In this survey, we provide insights into all the relevant work done in the past under various taxonomies, along with possible opportunities to address the challenges.
Fig. 1: An Overview of Neurosymbolic RL Process
The following is the document’s structure: Section II provides an overview of milestones in the AI field from its inception to the present day. In Section III, we present an overview of Neurosymbolic AI and RL, covering relevant literature and significant research findings. Section IV is dedicated to Neurosymbolic RL, including workable architectures and requirements. In Section V, we summarize notable research in Neurosymbolic RL under various headings. In Section VI, we discuss opportunities that have emerged from Neurosymbolic RL. Section VII is devoted to the challenges of implementing proposed Neurosymbolic RL applications. We identify the obstacles and challenges that may arise. Finally, in Section VIII, we offer concluding remarks on our survey paper.