Dimensionality reduction in deep neural networks

Abstract

Deep Neural Networks (DNNs) are powerful Artificial Intelligence (AI) techniques that enable machines to learn complex patterns from high-dimensional data. Due to the increasing complexity of data points and parameters in the model, DNNs experience the curse of dimensionality, where the model's efficiency exponentially decreases as the dimensionality increases. To solve this, dimensionality reduction techniques are often applied to feedforward DNNs to retain essential information and improve computational performance. However, the underlying mechanisms and explainability of these techniques remain insufficiently understood. This project aims to improve the explainability of feedforward DNNs and its usage of dimensionality reduction techniques. Specifically, we explore linear techniques such as Principal Component Analysis (PCA) and Singular Value Decomposition (SVD), alongside the nonlinear t-distributed Stochastic Neighbour Embedding (t-SNE), to evaluate how the feature representations evolve across network layers. We further assess the separability and structure of these representations using clustering methods including k-means algorithms and hierarchical clustering.

Experiments were conducted using the MNIST dataset, where we applied dimensionality reduction at each hidden layer and evaluated the compactness of intra-class representations and the separability of inter-class features using distance metrics, heatmaps, and visualization tools. PCA and SVD were used to estimate the number of principal components required to retain 0.95 of the total explained variance, while t-SNE offered more profound insights into non-linear manifold structures. Results show a progressive refinement of feature space through the layers, though certain digits remain ambiguously clustered.