Lightweight implementation of newly introduced “Differential Transformer”:
Lightweight implementation of newly introduced “Differential Transformer”:
Proposes differential attention mechanism which computes attention scores as a difference between two separate softmax attention maps thereby reducing noise in attention blocks. [[[Differential nanoGPT]]] :)
Code: https://github.com/Jaykef/ai-algorith...