VII Challenges
Neurosymbolic RL addresses a variety of issues that were previously challenges for DRL and has opened up new opportunities for researchers to develop novel methodologies. In this section, we outline a list of problems that are still prevalent with Neurosymbolic RL, including some that are specific to DRL and others that are more general research gaps.
VII-A Automated Generation of Symbolic Knowledge
Neurosymbolic RL relies on an environment where the agent can interact and receive rewards. Typically, these environments are represented by symbolic knowledge, as explained in the previous section. Symbolic knowledge encompasses both logic rules and knowledge graphs. While research into the automatic construction of non-logical symbolic part like knowledge graphs is relatively mature[112, 113, 114], the automatic learning of logic rules from data remains an underexplored area. Typically, domain experts manually construct the logic which is a time-consuming, laborious, and non-scalable process. Additionally, achieving end-to-end learning for rules that describe prior knowledge from data is a challenge for Neurosymbolic systems. Moreover, the inclusion of intricate logic, probabilistic relations, or diverse data sources adds further complexity to the problem. We contend that greater attention should be given to the comprehensive and automatic discovery of symbolic knowledge, not only from increasingly vast data sets but also from networks with rapidly expanding dimensionality.
VII-B Verification and Validation
Neurosymbolic RL models have gained popularity across multiple industries, with their size increasing rapidly to enable deployment in larger scenarios. These models have achieved state-of-the-art results and have provided a degree of reasoning and explainability. However, due to the relative novelty of this field, there is a lack of validation and verification methods for these models, which need to be addressed. For instance, despite AI surpassing humans in the game of Go in 2016, recent adversarial attacks on the models have exposed their weaknesses and led to humans defeating similar AI models in the game which have otherwise dominated grandmasters[^5]. This highlights the significant gap in the verification field, which requires extensive work to ensure that Neurosymbolic RL models are thoroughly validated and can be deployed without any flaws. Some work[93, 94, 115] has been initiated in this in this direction but prior work [116, 117] also need expansion so that they can be applied to Neurosymbolic RL domain.
VII-C Neurosymbolic RL Algorithms
The combination of neural, symbolic, reinforcement learning allows for a more comprehensive approach to problem-solving, as it enables the system to work with both numerical and symbolic data for RL. This provides a more powerful and flexible framework for learning, allowing for the integration of different types of knowledge and reasoning techniques for the agent.However, in order to effectively combine these fields, new learning algorithms need to be designed that can take advantage of the strengths of both symbolic and neural learning so as to be implemented in RL. The traditional reinforcement learning algorithms may not be accurate enough for Neurosymbolic learning, as they do not account for the complexities and nuances of symbolic reasoning.Therefore, new algorithms need to be optimized to work under the union of two sets of knowledge, leveraging the strengths of both neural and symbolic learning[118]. By doing so, researchers can develop more accurate and efficient learning algorithms that can be applied to a wide range of problems in fields such as natural language processing, robotics, and healthcare, among others.
VII-D Balancing Reasoning and Learning in RL
Neurosymbolic RL requires training the neural components using meaningful symbolic constraints and allowing the symbolic components to evolve with high-quality data-driven rules. However, transitioning between neural and symbolic components can lead to a loss of learning or reasoning power, which presents scalability challenges for the field. One crucial issue in Neurosymbolic RL is how to align symbolic specifications with representations learned using neural methods, known as the symbol grounding problem[119, 120]. This challenge is well-known in AI, but it is particularly complicated in Neurosymbolic RL, where symbolic and neural components can be interwoven in intricate ways.