Open Foundation Models: Scaling Laws and Generalisation

Talk-Titel: Open Foundation Models: Scaling Laws and Generalisation

Speaker: Dr. Jenia Jitsev

Abstract: To study transferable learning and generalisation, the derivation of reproducible scaling laws is crucial, as they not only predict model function and properties on unseen scales, but also have potential to enable systematic comparison of various learning procedures. The talk will discuss why open foundation models and datasets are essential for this research and highlight challenges in properly measuring generalisation.

Bio: Jenia Jitsev is co-founder and scientific lead of LAION e.V, the German non-profit research organization committed to research on open large-scale foundation models and datasets. He also leads Scalable Learning & Multi-Purpose AI (SLAMPAI) lab at Juelich Supercomputer Center, Research Center Juelich, Helmholtz Association, Germany and is a member of ELLIS. His background is in machine learning and neuroscience, aiming to understand learning as a generic process of incrementally building up a useful model of the surrounding world from available sensory observations and executed actions. His current research focus is on using scaling laws for measuring and understanding generalization and strong transfer in open foundation models. Jenia is most known for his work on open language-vision foundation models like openCLIP and open
datasets like LAION-400M/5B, Re-LAION, DataComp. Recently, he also has been studying reasoning and measuring generalization with works on open reasoning datasets/models OpenThouhts/OpenThinker and on discovering generalization weaknesses using AIW problems. Jenia coordinates acquisition of large-scale compute grants for conducting collaborative research on open foundation models across various supercomputing facilities, including EuroHPC. Using these resources, together with the community he is driving and democratizing research on scalable systems for generalist, transferable multi-modal learning, leading to foundation AI models capable of strong transfer and therefore easily adaptable to a broad range of desired tasks and hardware resource settings. He is currently involved in major efforts to establish strong open foundation model research and development track on EU grounds within EU projects openEuroLLM, ELLIOT and MINERVA. For his work, Dr. Jitsev received Best Paper Award at IJCNN 2012, Outstanding Paper Award at NeurIPS 2022 and Falling Walls Scientific Breakthrough of the Year 2023 Award.

freiburg.ai meets in the Nexus Lab. Averbis provides beers and bezels after the talk, and we invite you to stay for a chat.
Thanks for the support from Averbis!

This talk is organized by

Host and main contact: Jörg Franke