Neuro-symbolic (NeSy) AI aims at integrating neural learning from perceptual-level experience with reasoning through high-level knowledge. NeSy techniques can improve the robustness, generalisation capacity and interpretability of NNs, by encouraging their compliance with existing domain knowledge and constraints. NeSy systems are typically compositional, pairing a neural learning component with a symbolic, reasoning one. The former makes predictions from sub-symbolic input, while the latter reasons with the neural predictions through some background knowledge, to make high-level inferences. Consider for instance a vision model that extracts semantically meaningful features from images, coupled with a symbolic model that reasons with these features, in order to make high-level inferences for some downstream predictive task. Importantly, by making reasoning differentiable, the two components may be tightly integrated in a fashion that allows to train a NeSy system end-to-end, so that the neural predictions are aligned with the knowledge.
Most NeSy approaches assume that the symbolic part, i.e., the knowledge that is to be reasoned upon, is provided beforehand. In real-life applications, however, this is not always the case. Therefore, an important requirement in NeSy applications is the ability to extract high-level task semantics, in the form of symbolic knowledge, while training a network to make task-specific, low-level predictions. It is a challenging and under-explored problem, due to the fact that a “starting point” for the learning process is missing. A starting point could either be some concrete background knowledge, as in standard NeSy settings, or a pre-trained (or partially trained) network.
The purpose of this thesis is to explore techniques for mitigating difficulties in NeSy knowledge extraction via e.g. using NeSy training methods for simultaneously refining some crude, initial knowledge, while fine-tuning pre-trained networks w.r.t. the knowledge, such as foundation models, vision language models etc. Additional techniques for obtaining intermediate supervision, such as active learning and data programming techniques could be explored.
The project requires good knowledge of Python programming and a solid background on deep learning and knowledge representation & reasoning. The developed techniques may be applied on knowledge extraction for challenging applications, including autonomous driving, robot navigation and disease progression monitoring.
Bibliography:
Marra, G., Dumancic, S., Manhaeve, R. and De Raedt, L. From statistical relational to neurosymbolic artificial intelligence: A survey. Artificial Intelligence, 2024
Cunnington, D., Law, M., Lobo, J., & Russo, A., The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning. arXiv preprint arXiv:2402.01889, 2024
Liu, A., Xu, H., Van den Broeck, G., & Liang, Y., Out-of-distribution generalization by neural-symbolic joint training, AAAI, 2023