Explaining uncertainty (tabular data)

Abstract

Quantifying model confidence is essential for deploying neural networks in high-stakes decision-making. This study introduces a novel approach to uncertainty estimation, leveraging reconstruction error as a measure of prediction confidence. Rather than relying solely on probabilistic outputs, we propose a framework where a trained Multi-Layer Perceptron (MLP) model serves as a predictive model and a hidden layer’s output serves as a feature extractor. This hidden layer output acts as an encoder and a separate decoder is trained to reconstruct the original inputs. The error between the input and its reconstruction provides an uncertainty estimate, where higher reconstruction errors indicate lower model confidence. This methodology offers a systematic way to assess the reliability of neural network predictions, enhancing interpretability and robustness. We outline the training process and evaluate the model and correlation between uncertainty estimates and model prediction errors.