Accelerated Scan: Scalable Recurrence For Language Models - Volodymyr Kyrylov
Accelerated Scan is a library for high performance computation of first order recurrence relations, commonly found in state space models and recurrent language models. Unlike Transformers, it scales subquadratically in the sequence length and will allow to process sequences of lengths in the order of millions of tokens. I will introduce the implementation of the package and discuss paths to scale out.