Using mutual information to evaluate the generalization capability of deep learning neural networks

Abstract

There is a need to better understand how generalization works in a deep learning model. The goal of this paper is to provide a clearer view of the black box called neural network. This is done by using information theory to compute the flow of information within a network. The proposed framework uses an indicator that computes the mutual information of all hidden layers within the deep learning model. The indicator represents the predictive capabilities of the neural network. The evolution of the indicator provides another level of analysis with regards to the generalization capabilities. By using information theory, we can express the flow of information within a previously unseen black box. The framework provides the capability to analyse a deep learning model. It is a conceptual platform where users can perform analysis using the functions provided. Functions include computing mutual information, indicator usage, and visualization. Experiments were conducted using different methods to compute mutual information and its effects on deep learning models. It is found that our method based on the indicator overcome the shortcomings of using non-linear information bottleneck objective function. Our method computes the average mutual information of all hidden layers and this produces a better estimate as compared to the objective function which computes the mutual information of an intermediate representation. The advantage of the proposed framework includes that it is not restricted to a certain kind of neural network. Furthermore, it uses probability distribution functions which means the framework does not rely on the presence of the actual dataset. The focus of this paper is the use rather than the computation of mutual information. Therefore, methods such as non-linear information bottleneck or a neural network trained to estimate the mutual information of given datasets can be used to compute mutual information. The framework also provides a general solution to observe the learning process of a neural network and it is available at a public repository