Infini attention and Infini Transformer
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in this approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. On long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs, Infini-Transformer outperforms other methods. It introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
In this video, I talk about the following: What is Infini-Attention and how does Infini-Transformer work? How does Infini-Transformer perform?
For more details, please look at https://arxiv.org/pdf/2404.07143
Munkhdalai, Tsendsuren, Manaal Faruqui, and Siddharth Gopal. "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention." arXiv:2404.07143 (2024).