Using group theory to explore the space of positional encodings for attention

Attention is a computational primitive at the core of modern language models, allowing internal representations to reference and influence each other. It’s how these models handle sequential data in the first place.