Predicting MS Powerpoint mouse/keyboard actions

Abstract

This project explores the application of Generative Pre-trained Transformer (GPT) models, specifically GPT-2 and GPT-3, for predicting the textual instructions corresponding to user actions in Microsoft PowerPoint, such as mouse movements and keyboard inputs. Through extensive experimentation and implementation, we were able to observe how soft prompting with GPT-2 and in-context learning with GPT-3 exceed baseline performance established through hyperparameter tuning of the GPT-2 model. This achievement is particularly notable in two domains: the prediction of user intentions and the prediction of procedural instructions. Hence, this study underscores the efficacy of these techniques in augmenting the capabilities of the employed models. By illustrating the potential of AI-driven solutions to streamline interactions with software applications, this work sets a foundation for a shift in user experience within productivity tools, driven by seamless, natural language commands.