Self-aware and self-correction prediction

Abstract

Neural networks sometimes struggle to predict results with high accuracy in the presence of uncertainty and volatility. This study introduces a bias correction model that leverages an autocoder’s reconstruction errors to improve predictions in two distinct tasks: binary classification, and time series forecasting. We explored two datasets: a diabetes dataset for binary classification, and a SPY price history dataset for time series forecasting. On the diabetes dataset, a bias correction model was trained to enhance the F1-score of a multi-layer perceptron (MLP), particularly by reducing the number of false negatives, which is especially important in medical fields where missing a diagnosis could lead to severe consequences. With the SPY price history dataset, the bias correction models were trained on three distinct market phases, aimed to improve the prediction accuracy of a long short-term memory. Their performances were then evaluated separately. We observed greater improvements in periods where the market was relatively stable, and more limited gains when the market was more volatile. To further improve the effectiveness of the bias correction model, we incorporated a SHAP-based feature scaling method to emphasize the key features that affected the performance of the model. The result was a much better performance, even in periods of high market volatility. Future work could explore alternative feature selection methods, such as Recursive Feature Elimination (RFE) and LASSO regression to further optimize the bias correction model.