I Spy: An Interactive Game-Based Approach to Multimodal Robot Learning

Printer-friendly versionSend by email
Conference Proceedings (fully refereed)
Natalie Parde, Michalis Papakostas, Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, and Rodney D. Nielsen
Teaching robots about objects in their environment requires a multimodal correlation of images and linguistic descriptions to build complete feature and object models. These models can be created manually by collecting images and related keywords and presenting the pairings to robots, but doing so is tedious and unnatural. This work abstracts the problem of training robots to learn about the world around them by introducing I Spy, an interactive dialogue- and vision-based game in which players place objects in front of a humanoid robot and challenge it to guess which object they have in mind. The robot gradually learns about the objects and the features which describe them through repeated games, by updating its knowledge with newly captured training images. This paper details I Spy's learning and gaming processes, describes the approaches taken to extract information from multiple modalities both before and during gameplay, and finally discusses the results of a study designed to evaluate the game's model accuracy over time, its overall performance, and its appeal to human players.
Software and Knowledge Engineering Laboratory (SKEL)
Conference Short Name: 
AAAI 2015
Conference Full Name: 
Twenty-Ninth AAAI Conference on Artificial Intelligence
Conference Country: 
US:United States
Conference City: 
Austin Texas
Conference Venue: 
Hyatt Regency Austin
Conference Date(s): 
Sun, 25/01/2015 - Fri, 30/01/2015
Conference Level: 
AAAI Workshop on Knowledge, Skill, and Behavior Transfer in Autonomous Robots

© 2018 - Institute of Informatics and Telecommunications | National Centre for Scientific Research "Demokritos"