In the current AI landscape, the significance of trustworthy AI assets has grown exponentially as artificial intelligence permeates various aspects of our lives. Ensuring the utilisation of trustworthy AI systems is imperative for mitigating potential risks, biases, and ethical dilemmas. However, estimating the level of trustworthiness in an AI asset remains a complex undertaking due to the multifaceted nature of trustworthiness itself. Currently, this estimation primarily relies on self-assessment tools and frameworks, such as the ALTAI framework. While these tools offer valuable guidelines, it is important to acknowledge that they are subject to subjective judgments and may not fully encompass the breadth of trustworthiness. Hence, continuous research and development of more robust evaluation methodologies are necessary to enhance the assessment of trustworthy AI systems.
This work will study trustworthiness of AI assets within the European AI-on-Demand platform. With an intent to embrace a broader approach, it will potentially explore both algorithmic methods like link analysis, PageRank, and HITS, and non-algorithmic strategies, such as self-assessments, expert evaluations and crowd-sourced assessments. It will aim to underscore the importance of established AI trustworthiness methodologies and will incorporate principles of AI trustworthiness frameworks, such as the ALTAI framework.
Students will be required to explore and develop various strategies. In the case of algorithmic strategies, this will involve creating code that can assess the connectivity and importance of AI assets using algorithms like PageRank. Meanwhile, non-algorithmic strategies could involve creating tools to aid the effective self-assessment for AI asset creators, or crowd-sourced assessments. Considering the rich diversity of the AI landscape, it will be critical to consider a hybrid approach that combines algorithmic assessment with human judgement to foster nuanced evaluations.
This work will also place significant emphasis on creating appropriate experimental dataset to evaluate the various strategies being explored. This dataset will be created from a selection of AI assets available on the European AI-on-Demand platform, with the chosen assets providing a diverse cross-section of the AI landscape. The metadata to be considered will include adoption indicators, user reviews, the reputation of the asset creator, and usage statistics, and user ratings among others. These metadata will be indicative of the reliability and relevance of the AI assets, providing critical information for their evaluation.
This work will specifically aim at:
● building understanding AI asset metadata and the European AI-on-Demand platform
● studying core aspects of trustworthiness for AI applications utilising algorithmic and non-algorithmic methods
● creating datasets and ground truth for evaluating different algorithms for trustworthiness
● combining link based and other algorithms with qualitative trustworthiness assessment approaches to create more effective estimators
● explore the creation of an integrated trustworthiness scores for both algorithmic and non-algorithmic parts (i.e., by assigning weights to different factors based on their relevance and reliability, using machine learning techniques to adapt and improve over time)
Exact objectives will be discussed with the supervisory team.
Deliverables include:
– A dataset for trustworthiness estimation based on the European AI-on-Demand platform
– Alternative approaches towards algorithmic trustworthiness estimation
– Final MSc Thesis report
Required background and skills:
Python programming, algorithms, AI/ML knowledge
Other requirements:
All documents will be written in English.