[2024 Best AI Paper] MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation
This video was created using https://paperspeech.com. If you’d like to create explainer videos for your own papers, please visit the website!
Title: MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
Authors: Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan, Zhuoming Chen, Jinyuan Shi, Ian En-Hsu Yen, Beidi Chen
Abstract:
Large Language Models (LLMs) have become more prevalent in long-context
applications such as interactive chatbots, document analysis, and agent
workflows, but it is challenging to serve long-context requests with low
latency and high throughput. Speculative decoding (SD) is a widely used
technique to reduce latency without sacrificing performance but the
conventional wisdom suggests that its efficacy is limited to small batch sizes.
In MagicDec, we show that surprisingly SD can achieve speedup even for a high
throughput inference regime for moderate to long sequences. More interestingly,
an intelligent drafting strategy can achieve better speedup with increasing
batch size based on our rigorous analysis. MagicDec first identifies the
bottleneck shifts with increasing batch size and sequence length, and uses
these insights to deploy speculative decoding more effectively for high
throughput inference. Then, it leverages draft models with sparse KV cache to
address the KV bottleneck that scales with both sequence length and batch size.
This finding underscores the broad applicability of speculative decoding in
long-context serving, as it can enhance throughput and reduce latency without
compromising accuracy. For moderate to long sequences, we demonstrate up to 2x
speedup for LLaMA-2-7B-32K and 1.84x speedup for LLaMA-3.1-8B when serving
batch sizes ranging from 32 to 256 on 8 NVIDIA A100 GPUs. The code is available
at https://github.com/Infini-AI-Lab/Magi....