nanoGPT with Sigmoid Self-Attention
nanoGPT with Sigmoid Self-Attention
I couldn’t resist had to give it try:)
Some observations on M2:
SSA was ~5-10% faster in training with similar final loss values, slightly less coherent text generation, marginally higher perplexity, and lower memory usage compared to softmax.
Code: https://github.com/Jaykef/ai-algorith...
#ai #llm #gpt