Writing in the Margins: Better LLM Inference Pattern for Long Context Retrieval
In this video, we explain the Writing in the Margins (WiM) method, introduced in a recent research paper, titled: "Writing in the Margins: Better Inference Pattern for Long Context Retrieval".
With WiM, the researchers were able to achieve significant performance improvement on long input sequences, for off-the-shelf large language models (LLMs) such as Phi-3, Qwen2 and Llama-3.1.
How it works?
As part of the LLM inference process, the WiM method feed the input context to the LLMs by chunks, rather than all at once. And when each chunk is being processed, the LLM is also instructed to generate a note about the information in the current chunk. Finally, both the context, and the notes (which we refers as the margins), are available for the LLM to come up with the final response.
Paper page - https://www.arxiv.org/abs/2408.14906
Code - https://github.com/writer/writing-in-...
-----------------------------------------------------------------------------------------------
✉️ Join the newsletter - https://aipapersacademy.com/newsletter/
👍 Please like & subscribe if you enjoy this content
-----------------------------------------------------------------------------------------------
Chapters:
0:00 Introduction
0:58 Writing in the Margins (WiM)
3:15 WiM Example
3:53 WiM Results