contact: nmanginas@iit.demokritos.gr
Neuro-symbolic (NeSy) systems aim to combine neural learning and symbolic reasoning. They, usually, work compositionally. A neural network extracts a set of concepts from some complex input, e.g. an image, and reasoning is then performed on the neural network outputs towards a target query. For instance, consider a simple scenario in which an image of a traffic light is given, from which the neural network aims to extract its color, i.e. red, green or yellow. A rule is then provided which dictates that break should be predicted if the traffic light is red but not otherwise.
break ← color(traffic_light, red).
Reasoning shortcuts are a recently identified flaw in such neuro-symbolic systems, which hinders their explainability and trustworthiness. An example of a reasoning shortcut is that when given an input of a traffic light that is not red, whether the neural network predicts the color yellow or green the output of the rule will be the same, i.e. break will be false. This means the neural network might learn to detect green lights as yellow and yellow lights as green, or both green and yellow lights as green, e.t.c. As long as the output of the rule remains the same the NeSy system will perform well in predicting the target, i.e. when to break. However, when prompted to provide an explanation for why it didn’t break on an image where the traffic light is yellow, the system might reply because the traffic light is green. The phenomenon of reasoning shortcuts therefore reduces the interpretability of NeSy systems, along with several other drawbacks it introduces.
The student will: a) become familiar with basic NeSy systems, b) theoretically analyze the presence of reasoning shortcuts, c) understand state of the art mitigation strategies and d) develop new methods for avoiding reasoning shortcuts.
References:
[1] Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
[2] bears: make neuro-symbolic systems aware of their reasoning shortcuts
[3] A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts