Are vision language models multimodal learners?

Abstract

Since the release of accessible vision language models (VLMs) such as GPT-4V and Gemini Pro in 2023, scholars have envisaged utilizing these artificial intelligence (AI) models to widely support instructors and learners. Particularly, their capability to simultaneously process visual and textual data and yield subsequent information is considered one of the most important features of these user-friendly VLMs. This capability is significant as human cognition benefits from multimodality, which has called for teaching, learning, and evaluation to be conducted in more diverse, sophisticated, and constructive ways. However, these multimodal educational practices are yet to be realized in everyday classrooms, while the integration of AI promises to facilitate this transformation.

In this talk, we will review the hypothesized parallelism between humans and VLMs as multimodal learners and its implications for the potential role of AI models in future education. Additionally, we will discuss the limitations, challenges, and possible remedies to effectively integrate these models into educational settings.