Available Soon
Benjamin Busam, Technical University of Munich, Germany
Available Soon
Carol O'Sullivan, Trinity College Dublin, Ireland
Training, Reading and Steering Multimodal Foundation Models
Lorenzo Baraldi, Università degli Studi di Modena e Reggio Emilia, Italy
Short Bio
Carol O'Sullivan is the Professor of Visual Computing in Trinity College Dublin and head of the Graphics, Vision and Visualization (GV2) research group. From 2013-2016 she was a Senior Research Scientist at Disney Research in Los Angeles and also spent a year’s sabbatical as a Visiting Professor in Seoul National University from 2012-2013. She joined TCD in 1997 and served as the Dean of Graduate Studies from Jul'2007 to Jul'2010. Her research interests include Graphics & Perception, Computer Animation, Crowd and Human simulation. She was co-Editor in Chief for the ACM Transations on Applied Perception (TAP) for six years. Carol has been a member of many international program committees, reviewer for various journals, and served many times on the papers committees for the ACM SIGGRAPH and Eurographics conferences. She has chaired several conferences and workshops and is currently serving as program co-chair for Intelligent Virtual Agents (IVA 2017) and Motion in Games (MIG 2017). Prior to her PhD studies, she spent several years in industry working in Software Development. She was elected a fellow of Trinity College for significant research achievement in 2003 and of the European Association for Computer Graphics (Eurographics) in 2007.
Short Bio
Lorenzo Baraldi is Associate Professor at the "Enzo Ferrari" Department of Engineering of the University of Modena and Reggio Emilia, where he is part of AImageLab. His research is on multimodal foundation models with a focus on retrieval-augmented generation, visual grounding, and the safety and trustworthiness of generative models. He is an ELLIS Scholar and coordinates the ELLIS Unit in Modena. Within the Horizon Europe project ELLIOT he coordinates the large-scale pretraining of open multimodal foundation models, and he leads UNIMORE's contribution to several European and national initiatives at the intersection of AI and high-performance computing, among them MINERVA (EuroHPC), ELIAS, ELLE and IT4LIA, the Italian AI Factory. He serves as Associate Editor for Pattern Recognition, Pattern Recognition Letters and Computer Vision and Image Understanding, and regularly as Area Chair for CVPR, ICCV, ECCV, AAAI and ACM Multimedia. He has co-authored more than 180 publications, which have collected over 9,000 citations (h-index 44). Earlier in his career, in 2017, he worked at the Facebook AI Research laboratory in Paris under the supervision of Hervé Jégou. During that time, he worked on the development of a video-matching algorithm that was adopted in production on the social network to detect abusive content.
Abstract
Multimodal foundation models are now trained at a scale that only a few organisations, and a handful of public supercomputers, can afford. Yet scale alone does not fix their most persistent failures: they misread fine visual details, answer from memorised associations rather than from evidence, and inherit harmful concepts from web-scale data. This keynote argues that the internal representation of these models is the place where such failures can be diagnosed and corrected, often without retraining the model. I will start from the training side, drawing on ELLIOT, a Horizon Europe project in which European research groups and EuroHPC supercomputing centres jointly pretrain open multimodal foundation models, and discuss what scale buys and where it stops helping. I will then look inside the models along four directions. First, reading the representation: the attention heads of a multimodal LLM differ widely in how well they align with a vision encoder, and aligning the least aligned ones gives the largest gains. Second, enriching it: retrieving external multimodal knowledge, and reasoning over which retrieved evidence is actually relevant, grounds the answers to knowledge-intensive visual questions. Third, steering it: safety can be edited directly in the embedding space, and a safety direction estimated on one generator can be transferred to other image and video generators. Fourth, reading out what a generator draws: in diffusion transformers, a few channels with massive activations carry the semantics of the whole image and can be transplanted across prompts without training. Together, these results suggest treating representations as the interface through which large models can be controlled and made reliable, for recognition and generation alike.