Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/171932
Title: Collecting and annotating videos that teach MS PowerPoint
Authors: Tan, Isaac Jun Hong
Keywords: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
Issue Date: 2023
Publisher: Nanyang Technological University
Source: Tan, I. J. H. (2023). Collecting and annotating videos that teach MS PowerPoint. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/171932
Abstract: The central aim of this project is to generate a comprehensive dataset for training an artificial intelligence (AI) that is able to operate Microsoft PowerPoint autonomously. This project encompasses several different phases: Starting with the identification of videos that teach Microsoft PowerPoint following which we will download the identified videos using Jupyter Notebook with the help of the Pytube library. This is followed by the transcribing of videos that lack closed captions with the Whisper Model. Following this, the annotation process is then executed whereby the keystroke and the mouse clicks are then labeled using Sequence labeling in Doccano. The project then transits into the model training phase where both T5 and FLAN-T5 neural network models are experimented on for their ability to interpret and translate narrated instructions into corresponding mouse and keyboard actions to decide which model would achieve the better performance. Given the limitations of YouTube’s dataset, data augmentation techniques were employed using ChatGPT to improve model training.
URI: https://hdl.handle.net/10356/171932
Schools: School of Computer Science and Engineering 
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
Isaac_Tan_Jun_Hong_Final_Year_Project_Report.pdf
  Restricted Access
Undergraduate project report1.22 MBAdobe PDFView/Open

Page view(s)

106
Updated on Jul 10, 2024

Download(s)

10
Updated on Jul 10, 2024

Google ScholarTM

Check

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.