World Knowledge in the Time of Large Models

Videos
SILO
UW-Madison
VLM
LLM
LMM
Multimodal learning
Foundation models
This talk will discuss the massive shift that has come about in the vision and ML community as a result of the large pre-trained language and language and vision models such as Flamingo, GPT-4, and other models.
Presenter
Affiliation

Kenneth Marino, PhD

Research Scientist, Google DeepMind

Date

November 22, 2023

Summary from SILO website

Bio: Kenneth Marino is Research Scientist at Google DeepMind in NYC, focusing on improving knowledge-based systems such as retrieval and information extraction as well as embodied reasoning with language. He graduated in 2021 from Carnegie Mellon University advised by Abhinav Gupta, where his thesis focused on incorporating knowledge into embodied systems. He has an adjunct appointment at Columbia University where he teaches a class focused on the impact of datasets on machine learning and how to collect good datasets. He received his undergraduate degree from the Georgia Institute of Technology where he studied Computer Engineering and Computer Science.

Abstract: This talk will discuss the massive shift that has come about in the vision and ML community as a result of the large pre-trained language and language and vision models such as Flamingo, GPT-4, and other models. We begin by looking at the work on knowledge-based systems in CV and robotics before the large model revolution and discuss the impact it had. This impact can be broken down into three areas in which world knowledge should be studied in the context of these new models: evaluation, harnessing large models, and building outside knowledge. First, evaluating world knowledge is even more important as the large model revolution gives more easy access to world knowledge. Next, we discuss recent work in harnessing models such as Flamingo and Chinchilla for visual and procedural knowledge. Finally, the talk discusses how, by focusing on knowledge acquisition as an agent-centric problem, we can make developments in retrieving and collecting world knowledge.

Jump to section