The AI-powered Entity Audit
- How Machines Decide What Your Page Is About
- Internal Linking on Entity Overlap
- Fixing Clarity: The Highest-Yield Edits
Generative AI & LLMs
In shortThe technique that lets a neural network weigh how much each word in a text matters to every other word, so it reads words in context.
The attention mechanism is the part of a neural network that decides which other words in a text matter most for understanding each word. Instead of reading strictly left to right, the model looks at every word in relation to all the others and gives the important ones more weight. It is the core idea behind the Transformer architecture, and therefore behind BERT and modern large language models.
Each word (strictly, each token) starts as an embedding, a list of numbers. For every token, attention compares it with every other token in the input and calculates a score for how relevant each one is. Those scores become weights, and the token’s new representation is a weighted blend of the others. So “bank” in “river bank” ends up with a different representation from “bank” in “bank account”, because different neighbours received the most attention.
Transformers use self-attention (a text attending to itself) and run several attention “heads” in parallel, each free to pick up a different kind of relationship. The 2017 paper “Attention Is All You Need” showed that a network based solely on attention, with no recurrence, could outperform earlier translation models while being more parallelisable and faster to train. That efficiency is what made training very large language models practical.
You will rarely configure attention yourself, but it explains behaviour you see every day:
Take the query “running shoes for flat feet not nike”. A bag-of-words model sees “nike” as a keyword match. An attention-based model connects “not” to “nike” and “for” to “flat feet”, so it understands the searcher wants supportive shoes from other brands. Keyword tools that rely on exact matches will group this query with Nike pages; embedding-based clustering is more likely to group it correctly.