Self-attention — Self-attention is a form of attention in which every word in a passage looks at every other word to understand context. It lets a model see that bank means something different in river bank and bank account. It is computed in parallel, which makes it fast on modern chips. It is the central operation inside transformers.

Added October 21, 2026