ML and systems co-design for resource-efficient LLM inference serving

Abstract

This project investigates the co-design of machine learning and systems-level techniques to enable resource-efficient inference serving of large language models (LLMs). We conduct a detailed analysis of the trade-offs between accuracy and performance (latency, memory, throughput) for a variety of model-level optimizations, including a modified version of Skeleton-of-Thought (SoT) prompting, prompt compression using LLMLingua, model quantization, and Key-Value Cache quantization. Through systematic experimentation and analytical modeling, we identify efficient configurations that minimize resource usage while maintaining output quality close to that of large models.

In addition to understanding the efficiency-accuracy trade-offs of the earlier mentioned model-level optimizations, the key contributions of this project also include the implementation of system components such as parallel expansion handlers, compression routers, and output routers—designed to integrate the evaluated optimizations into an adaptive, modular serving system. These components are built on top of the vLLM engine to leverage modern serving capabilities like continuous batching and paged attention.

Together, our findings and implementations lay the foundation for a complete LLM serving stack that can dynamically adapt to input characteristics and system load by leveraging various machine learning optimizations. The full integration of these components into a complete system remains ongoing work.