Developing a graphene Q&A chatbot using retrieval augmented generation (RAG)

Abstract

Graphene synthesis is a rapidly growing market with various methods for different applications. However, the mass production of high-quality graphene that is cost-effective and environmentally sustainable has not been established commercially. Current graphene synthesis techniques also face issues related to reproducibility. Recently, the proliferation of artificial intelligence (AI) with ever-evolving large language models (LLMs), along with the emergence of the Retrieval Augmented Generation (RAG) approach, has demonstrated significant abilities to produce natural responses with a vast amount of knowledge. Therefore, there is an interest in combining the database of graphene synthesis with AI to remarkably assist in the research process. This experimental study tested the use of UMAP visualizations to determine the optimal chunk size and overlap. Subsequently, two LLMs, the DRAGON Deci-7B LLM and the DRAGON Mistral-7B LLM, were tested within a RAG question-answering chatbot architecture. The chatbots were then further evaluated with two advanced retrieval methods: the parent document retriever and the ensemble retriever. The chatbots were evaluated by RAGAs, a performance metric framework, with ChatGPT as a benchmark using a synthetic dataset of 10 questions and corresponding ground truths. Human evaluation was also conducted by manually inputting a user prompt into the chatbots and analysing the response generated. In summary, through LLM evaluations with ChatGPT, the optimal chatbot developed in this study utilized the DRAGON Mistral-7B LLM with the parent document retrieval method, with an embedded chunk size of 256 tokens and a 10% overlap. However, human evaluations raised concerns with regards to the actual useability of the chatbot. Further troubleshooting and refinement would be necessary, but this was constrained by the costs associated with the project.