Stock price predictability and the business cycle via machine learning

Abstract

The negative effects of data shifts on machine learning (ML) model performance have been extensively characterized in numerous applications of ML (e.g. natural language processing). However, few studies are exploring the data-shifting effects of the business cycle, particularly in the context of stock price forecasting. Hence, this research studies how the business cycle affects stock price forecasting and explores possible mitigating measures. Using the S&P 500 index, we begin by documenting a stylized fact that most prediction models perform worse during the US recessions, which is public information dated by the National Bureau of Economic Research (NBER). We further investigated those models that did not generate worse predictions during recessions. They were observed to occur during the sub-periods of the oil crisis in the late 70s when the interest rate reached a historical peak. Interestingly in these sub-periods, we discover that the test volatility of realized stock returns has little difference between the recession and non-recession days. That implies the better prediction performance observed in the 70s is not the merit of the ML methods. Instead, it is perhaps the effective monetary policy that stabilized the stock market during those sub-periods. Additionally, we show that the inclusion of recession data in the training set and the incorporation of the risk-free rate and VIX index as input variables did not significantly improve recession performance. To manage the negative implications of recession volatility, we proposed an auto-encoder-inspired model architecture capable of generating confidence scores consistent with true model performance. Comparing the confidence model's recession predictions with and without confidence-based decision-making, we demonstrated that prediction errors during recessions could be lowered with a carefully selected confidence threshold. Besides this measure, we also recommend ML practitioners evaluate their models on both recession and expansion data. Future work could explore other methods of showcasing the negative impacts of a recession and other methods of relieving its effects.