LLMLingua: Compressing Prompts for Accelerated Inference of LLMs
LLMs have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, LLMLingua is a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. Experiments and analysis are done over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23. They show that LLMLingua yields state-of-the-art performance and allows for up to 20x compression with little performance loss.
In this video, I talk about the following: How does LLMLingua compress prompts? How does LLMLingua perform? Case study on GSM8K math reasoning prompt compression. Can we recover original prompts compressed by LLMLingua?
For more details, please look at https://llmlingua.com/ and https://arxiv.org/pdf/2310.05736
Jiang, Huiqiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models." In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13358-13376. 2023.