YOCO: Decoder-Decoder Architectures for LLMs
YOCO is a decoder-decoder architecture for large language models, which only caches key-value pairs once. It consists of two components, i.e., a crossdecoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. YOCO extended to 1M context length leads to near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes.
In this video, I talk about the following: What is the YOCO architecture?How does the YOCO model perform?
For more details, please look at https://arxiv.org/pdf/2405.05254 and https://aka.ms/YOCO
Sun, Yutao, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. "You Only Cache Once: Decoder-Decoder Architectures for Language Models." arXiv preprint arXiv:2405.05254 (2024).