The goal of this project is to propose novel machine learning methods across a multitude of problems such as classification, clustering and summarization. We investigate two important aspects of learning systems. Firstly, we look into data representations for different modalities, as well as multi-modal data. We compare different approaches and fusion schemes evaluating performance, expressiveness and applicability per problem. Secondly, we examine the architecture, training and performance of modular learning models such as neural networks and their expansion with nontraditional components that directly take into account statistical and/orsemantic information of their input. The findings of our research will be merged into a novel learning pipeline bearing advantages to the current state of the art. This work will enable the wealth of semantic and statistical research to complement the potential of neural networks and support a holistic view of learning, beyond the borders of a specific learning implementation.