Transforming assessment with LLM and generative AI: impacts and challenges

Abstract

The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have impacted our ways of doing many things. Given its central role in accrediting knowledge and skills, assessment positions itself at the forefront of these impacts. Applying cutting-edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely, these innovations raise significant concerns regarding validity, reliability, transparency, fairness, equity, and test security, necessitating careful thinking when applying them in assessments. In this talk, I will discuss the impacts and implications of LLMs and generative AI on critical dimensions of assessment with example use cases and highlight the challenges that call for a community effort to address.