Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Paper: https://arxiv.org/abs/2404.07143
My notes: https://drive.google.com/file/d/1plWJ...
00:00 Intro
07:17 Model intuition
11:00 Memory retrieval operation
16:29 Hidden state updates
21:58 Delta update
24:10 Is it causal?
25:26 Combining local attention and RNN
27:26 Results
30:25 Sampling and Conclusion