Explore the fine-tuning of pre-trained LLM models to improve the quality of abstracts for long patent descriptions

Abstract

Generating abstracts for long patent descriptions is a challenging task. Many patent descriptions exceed 10,000 words in length, while the maximum token limit for most standard text summary models is typically only 512, 2048, or 4096 tokens. As a result, patent descriptions often surpass the token limits of these models. This project focuses on fine-tuning large language models (LLM) to effectively summarize lengthy patent descriptions with high quality. It focuses on overcoming the challenges of generating concise abstracts for lengthy patent descriptions by selecting and adapting suitable models for extended text processing. It investigates the performance of Qwen, InterLM, LLaMA, and RWKV models. Through fine-tuning and comparison, the project successfully demonstrated that effective long-text summarization can be achieved. Techniques such as LoRA and position encoding extrapolation were employed to address hardware limitations (using a single graphics card). Additionally, enhancements to the models, including improvements in the attention mechanism, have made them better suited for handling long text tasks.