[2024 Best AI Paper] Contextual Position Encoding: Learning to Count What's Important
Join Discord to tell us your ideas about the video: / discord
Title: Contextual Position Encoding: Learning to Count What's Important
Authors: Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar
Abstract:
The attention mechanism is a critical component of Large Language Models
(LLMs) that allows tokens in a sequence to interact with each other, but is
order-invariant. Incorporating position encoding (PE) makes it possible to
address by position, such as attending to the i-th token. However, current PE
methods use token counts to derive position, and thus cannot generalize to
higher levels of abstraction, such as attending to the i-th sentence. In this
paper, we propose a new position encoding method, Contextual Position Encoding
(CoPE), that allows positions to be conditioned on context by incrementing
position only on certain tokens determined by the model. This allows more
general position addressing such as attending to the $i$-th particular word,
noun, or sentence. We show that CoPE can solve the selective copy, counting and
Flip-Flop tasks where popular position embeddings fail, and improves perplexity
on language modeling and coding tasks.