LLM-based learning companion & co-pilot – a video to text approach

Abstract

The growing integration of large language models (LLMs) in education has paved the way for innovations such as video-based learning assistants. This project explores the implementation of an LLM-based Learning Companion and Co-Pilot designed to complement a lecture video to facilitate student learning. The primary goal is to extract meaningful textual context from lecture videos, enabling the chatbot to respond effectively to students’ queries. Azure services, including Azure AI Video Indexer, Azure OpenAI, and Azure Cosmos DB, were utilized. Different Retrieval-Augmented Generation (RAG) implementations, including hybrid retrieval and LLM document transformation processes, were evaluated. In addition, a user interface was developed to provide students with an intuitive platform to access the chatbot and video content. The chatbot’s performance was assessed using the RAGAS (Retrieval Augmented Generation Assessment) evaluation framework and user testing among the College of Computing and Data Science (CCDS) students. The findings showed that the proposed RAG v3 implementation, using hybrid retrieval combined with the LLM document transformation process, improved the chatbot’s effectiveness and temporal understanding. However, it still faced challenges in handling complex and temporal queries, highlighting opportunities for future development.