Investigating large language model pruning techniques

Abstract

In recent years, the rapid development of large language models (LLMs) has significantly advanced the performance of various natural language processing (NLP) tasks. However, this progress has also introduced challenges related to the high cost of training, deployment, and model transfer. Model pruning, as an effective optimization technique, reduces model size by eliminating a portion of the model parameters. Among various pruning methods, structured block pruning is widely adopted in the context of LLMs due to its pruning efficiency. Nevertheless, its typical approach of merely zeroing out weights limits the model's representational capacity and often leads to substantial performance degradation. To address this issue, we propose a novel optimization strategy that integrates structured block pruning with knowledge distillation under a sparse lazy-loading framework. Specifically, after structured block pruning, the remaining dense-format tensors are converted into sparse-format tensors following the Block Sparse Row (BSR) layout, effectively reducing the memory footprint required to store the pruned LLM. Subsequently, the pruned model—equipped with the sparse lazy-loading mechanism—is distilled using the original, unpruned model as the teacher. This distillation process aims to recover the language modeling and natural language reasoning capabilities of the pruned model. Empirical results demonstrate that although the distilled sparse lazy-loaded structured block-pruned model exhibits slightly lower compression ratios compared to its non-distilled counterpart, it achieves significantly better compression efficiency than the original, unpruned model. Moreover, it shows superior performance in both language modeling and natural language inference tasks relative to the non-distilled pruned model, approaching the performance of the original model.