BERT (Bidirectional Encoder Representations from Transformers)

BERT (Bidirectional Encoder Representations from Transformers) is a language model published by Google researchers in 2018. It reads every word in the context of the words on both sides of it, so the same word gets a different numerical representation depending on the sentence. That contextual understanding made BERT the base for a generation of NLP tools SEOs use, from KeyBERT to BERTopic, and Google has used it in Search since 2019.

How it works

BERT is an encoder built on the transformer architecture. It splits text into sub-word tokens and passes them through stacked attention layers, where each token “looks at” every other token. It was pre-trained on unlabelled text (BooksCorpus and English Wikipedia) with two tasks: masked language modelling, where 15% of tokens are hidden and the model predicts them, and next-sentence prediction. The original paper released two sizes, BERT-Base (110 million parameters) and BERT-Large (340 million).

After pre-training, BERT is fine-tuned with a small labelled dataset for a specific job such as classification, named entity recognition or question answering. Because it is an encoder, it is built to understand text and produce embeddings, not to generate long answers the way GPT-style models do.

How it’s used in SEO and marketing

• Keyword extraction with KeyBERT, which compares the embedding of a document with the embeddings of candidate words and n-grams using cosine similarity.
• Topic modelling of page content with BERTopic, and keyword clustering with Sentence-BERT, a BERT variant tuned for sentence similarity.
• Search intent classification with fine-tuned BERT or DistilBERT models.
• Extractive summarisation, for example drafting meta descriptions by picking the most representative sentences on a page.
• Understanding Google itself: Google said in 2019 that BERT would help it better understand about one in ten English searches in the US, especially longer, conversational queries where small words like “to” change the meaning.

Example

Lazarina’s meta description tutorial began in 2021 with BERT extractive summarisation, then the most accessible way to summarise pages at scale. The 2026 refresh keeps that method alongside small LLMs: BERT picks the sentences that best represent the page, which gives you a draft grounded in your own copy rather than newly generated text.

Limitations and things to watch

• Inputs are capped at 512 tokens, so long pages must be chunked or truncated.
• Raw BERT token embeddings are not ideal for comparing whole sentences; Sentence-BERT style models were created to fix that, and BERTopic’s default embedding model is a sentence-transformers model rather than original BERT.
• The original model is English-only; use a multilingual variant for other languages.
• Fine-tuning and large-scale inference are much faster on a GPU (a free Colab GPU is usually enough for SEO-sized datasets).

Learn it on MLforSEO

• How to do keyword clustering with KeyBERT
• Topic Modelling for SEO with BERTopic
• How to Automatically Generate Meta Descriptions with Python: From BERT to LLMs
• Different ways to map keywords to topics with supervised and unsupervised ML
• Introduction to Machine Learning for SEO (course)