Enhancing financial market volatility prediction: a machine learning approach integrating sentiment analysis and macroeconomic indicators

Abstract

This report aims to explore the integration of sentiment-derived features with machine- learning (ML) algorithms – LightGBM (LGBM) and XGBoost to improve volatility forecasting. Financial market volatility forecasting continues to be a significant challenge in risk management and investment decision-making. In this study, sentiment data from Reddit posts, representing retail investor sentiment, and The Wall Street Journal, reflecting institutional perspectives alongside historical market index data from 2022 to 2024 were added into our ML models to assess their impact on volatility forecasting. Utilising these ML models, the study evaluates whether sentiment enhanced models offer additional predictive power beyond baseline models that rely solely on historical data and traditional econometric approaches, such as EGARCH. The results show that LGBM and XGBoost models consistently outperform EGARCH, demonstrating lower mean squared errors (MSE). Incorporating sentiment data from both Reddit and the Wall Street Journal with lagged historical data significantly improves predictive accuracy, achieving the lowest MSE for both Dow Jones and S&P 500 indexes. Additionally, while macroeconomic indicators contribute to model performance, they are less effective than historical market trends in short-term volatility forecasting. Diebold-Mariano test results along with a simulation study reinforces the statistical significance of machine learning models' enhanced accuracy over EGARCH. These findings strengthen the case for integrating machine learning techniques into volatility forecasting as viable alternatives to traditional econometric models in financial applications