Social media data mining : implementing a social media data mining pipeline for personality computing

Abstract

Social Media has been thoroughly integrated into the many facets of societies across the world, churning out vast quantities of valuable data that hides a multitude of insights. In recent years, many novel techniques and methods have been brought to light and made mainstream through open-source repositories. These cutting-edge tools have allowed applicants of the technology to rapidly produce a multitude of applications that extract insights from social media data.

Attempted here will be a social media data mining pipeline to perform automated personality assessment and evaluation. This pipeline consists of 5 stages in sequence; data collection, data transformation, data preprocessing, model execution and personality evaluation. To discover how best to implement each stage, exploratory analysis and experiments were conducted for familiarising with the materials and comparison’s sake respectively.

Primary to the pipeline is an analysis and classification on social media users’ personalities through analysing their historical timeline laced with their opinions, comments, ideas, and interactions. Each tweet will be analysed for its sentiment, emotion, and personality traits. Consulting the big-five personality trait model, behavioural classification using pre-built models, transformers and Zero-Shot classification will be used. Additionally, the pipeline will be tested by feeding thousands of tweets collected from Twitter using API scraping methods. The pipeline was then later deployed onto a web application as a proof of concept (PoC) implemented using Streamlit which also includes various visualisations and options for customising the pipeline.