RAGLens: a study on understanding, evaluating, and comparing retrieval-augmented generation techniques

Abstract

Artificial intelligence has been advancing unprecedentedly, and its capabilities have a massive potential to transform existing educational tools. Despite the capabilities and potential of large language models (LLMs), there is a significant downfall that impacts their widespread adoption: hallucination. LLMs may generate content that is not grounded in factual information. These hallucinated contents are complex for learners to detect when they do not have sufficient domain knowledge. Acquiring incorrect knowledge in the early phase can negatively impact education. Retrieval-augmented generation (RAG) is a technique that is posed to be the solution to this problem. RAG reduces the chance of hallucination by incorporating a knowledge base into the LLMs so that LLMs can retrieve relevant content to answer user queries. Ensuring that the answer generated is grounded in the knowledge base can minimize the chance of hallucination. However, RAG is not a one-size-fits-all solution. RAG is a framework that encompasses a wide range of techniques and variations. There is no single best RAG pipeline that guarantees the best result across all scenarios. The complexity of RAG makes it hard for developers to optimize their RAG applications conveniently and confidently. Hence, this project aims to investigate the current state-of-the-art RAG practices and techniques that generate more accurate outputs than the default RAG pipeline. In addition, the project will introduce an application that allows developers to explore and evaluate RAG techniques, helping them to identify the optimal RAG practices and methods according to their knowledge base and user queries.