Topic Modelling for SEO with BERTopic

If LDA is the classic that taught search to think in topics, BERTopic is the modern successor that does it with the semantic muscle of transformer embeddings. For most of the text-heavy work SEOs actually do, clustering keywords, reviews, titles and content, it is the approach I reach for first. This post explains what it is, how it works, walks through the actual code so you can run it yourself, and gives you the honest benefits and limitations so you know exactly when it is the right tool.

It also goes deep on the one thing that makes or breaks most topic modelling projects in SEO: short text. Titles, headings, anchor text, queries, reviews and social posts are where every topic model struggles, so I’ve included the eight reasons why, and what to do about each one.

What BERTopic is

BERTopic is a topic modelling technique, introduced by Maarten Grootendorst in his 2022 paper “BERTopic: Neural topic modeling with a class-based TF-IDF procedure”, that uses transformer embeddings (by default a BERT-style sentence-transformers model) and class-based TF-IDF to create dense, easily interpretable clusters while keeping the most important words in each topic’s description. Its defining feature is that it is built like Lego: you can pick and choose the model for each step, tokenise one way, cluster with HDBSCAN or K-means or BIRCH, swap in a model from Hugging Face. That modularity is what makes it far more versatile than LDA.

How BERTopic works

Like any topic model, it ends up identifying sets of words that form coherent themes. Under the hood, it reverse-engineers those themes through a clean pipeline.

It converts text into 384-dimensional embeddings from a compact sentence-transformer model in the BERT family that capture semantic relationships, uses UMAP to reduce those to a low-dimensional space, applies HDBSCAN to find dense clusters (each a potential topic), and then uses class-based TF-IDF to surface the most representative words per topic, refined with MMR for relevance. You can feed it your own topic labels or none at all, in which case it discovers topics automatically. In short, it is an embedding-based, density-based, unsupervised model that assigns one topic per document by default. Recent versions from its creator, Maarten Grootendorst, add zero-shot topic modelling, model merging and large language model support, which keep it at the front of the field. The 0.17 releases (2025) added a lightweight installation without UMAP and HDBSCAN, fast static embeddings through Model2Vec, and LiteLLM for topic labelling with almost any LLM.

(The 384 dimensions come from the default English embedding model, all-MiniLM-L6-v2. Swap in a bigger model and you get more dimensions.)

The class-based TF-IDF step is worth understanding, because it is what gives BERTopic its readable labels. Ordinary TF-IDF weighs how important a word is to a single document; the class-based version treats all the documents in a cluster as one big document and asks which words are distinctive to that cluster versus the others. The result is a topic description made of the words that genuinely set that topic apart, rather than generic high-frequency filler, which is a big part of why BERTopic’s topics tend to read so cleanly compared with older bag-of-words approaches.

Running BERTopic yourself: the code

Here is the part I want you to actually try. BERTopic is refreshingly little code to get going. In my Introduction to Machine Learning for SEO course you get the fully annotated Colab notebook, the demo dataset and a ready pipeline, but there is no reason you cannot work through the essentials on your own right now. Start with the simplest possible run on a list of documents (your pages, reviews, titles, whatever your corpus is).

Everything below was tested in October 2026 on BERTopic 0.17.4 and Python 3.12 (BERTopic no longer supports Python 3.9). The complete, runnable version, with a demo dataset, visualisations, exports and the short-text experiments from later in this post, is in the MLforSEO code hub.

!pip install bertopic
from bertopic import BERTopic
docs = df["content"].tolist()          # your pages, reviews or titles as a list of strings
topic_model = BERTopic(language="english", min_topic_size=10)
topics, probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()           # the topics it discovered

That is genuinely all you need for a first pass. The magic, and the reason I love this model, is in the “build your own Lego” step, where you swap the embedding model, the dimensionality reduction and the clustering to suit your data.

from sentence_transformers import SentenceTransformer
from umap import UMAP
from hdbscan import HDBSCAN
from sklearn.feature_extraction.text import CountVectorizer
from bertopic.vectorizers import ClassTfidfTransformer
from bertopic.representation import KeyBERTInspired
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")     # 384-dim embeddings
embeddings      = embedding_model.encode(docs, show_progress_bar=True)   # compute once, reuse
umap_model      = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric="cosine", random_state=42)
hdbscan_model   = HDBSCAN(min_cluster_size=30, min_samples=5, metric="euclidean", prediction_data=True)
vectorizer      = CountVectorizer(stop_words="english", ngram_range=(1, 2), min_df=2)
ctfidf_model    = ClassTfidfTransformer(reduce_frequent_words=True)   # down-weights topic-general words
topic_model = BERTopic(
    embedding_model      = embedding_model,
    umap_model           = umap_model,
    hdbscan_model        = hdbscan_model,
    vectorizer_model     = vectorizer,
    ctfidf_model         = ctfidf_model,
    representation_model = KeyBERTInspired(),   # cleaner topic labels
    calculate_probabilities = True,
    verbose = True,
)
topics, probs = topic_model.fit_transform(docs, embeddings)

Every one of those blocks is optional and swappable. Prefer K-means because you want a fixed number of topics? Pass it as the hdbscan_model. Working in another language? Swap the sentence transformer. That is the flexibility I like to call “building your Lego”, and it is what puts BERTopic a level above LDA.

Three small changes from the version of this snippet you may have seen before, all learned the hard way. Computing the embeddings once and passing them in means you can re-run the clustering with different settings in seconds, without re-embedding. Setting random_state on UMAP makes your topics reproducible from run to run. And the HDBSCAN settings matter more than anything else for the number of topics: on my demo data, min_cluster_size=15 lumped computer graphics and space into one topic, while min_cluster_size=30, min_samples=5 split them into ten cleaner topics. Tune these for your data.

What the output actually looks like

Once it has run, you inspect and visualise. The course demo deliberately uses the same small BBC news dataset (about 100 articles) as the course’s LDA and text classification demos, so you can compare the three approaches directly. BERTopic pulls out six coherent topics from it, family care and support, climate change, a Trump/election topic, and the Ukraine-Russia war among them.

topic_model.get_topic_info()
#    Topic  Count  Name
# 0     -1     11  -1_the_to_of_and                 <- outliers, no clear topic
# 1      0     28  0_ukraine_russia_putin_war
# 2      1     19  1_climate_emissions_energy
# 3      2     15  2_trump_election_republican
# 4      3     14  3_family_care_support_children
topic_model.get_topic(0)
# [('ukraine', 0.041), ('russia', 0.033), ('putin', 0.028), ('war', 0.021), ...]

Notice topic -1: that is the outlier bucket, documents HDBSCAN could not confidently place. It is normal, and it is one of the trade-offs we will get to. From here you visualise, reduce the topic count if it over-produced, and export each document’s topic.

topic_model.visualize_topics()      # interactive inter-topic distance map
topic_model.visualize_barchart()    # top words per topic
topic_model.visualize_hierarchy()   # see which topics could merge
topic_model.reduce_topics(docs, nr_topics=20)   # rein in an over-produced list
df["topic"] = topic_model.topics_   # tag every document (topics_ reflects any reduction)
df.to_csv("content_with_topics.csv", index=False)

That CSV, every page or review tagged with its dominant topic and the representative words, is the artefact you actually take into an internal-linking or content-audit project. A quick, important caveat: this is a very limited demonstration of what BERTopic can do. If you are going to use it seriously, the creator’s GitHub repository is a must-read, it is full of tips on scalability, coherence and customising the pipeline.

The public notebook uses a different demo (six groups from the 20 Newsgroups dataset, about 3,400 posts, each with a long body and a short subject line) because it is freely downloadable, and because the subject lines are perfect for the short-text experiments below. On the full posts it found ten topics (cars, guns politics, hockey, medicine, space, and a few graphics subtopics), with 8% of posts in the outlier bucket.

BERTopic versus LDA and the others

Choosing a topic model is really about your data and your constraints. Here is how the main options compare.

The headline differences: BERTopic and Top2Vec need minimal preprocessing and detect the number of topics automatically, which makes them strong on short texts like reviews, tweets and titles, exactly the data we tend to have. LDA and NMF need more preprocessing and you must predefine the topic count. And BERTopic is usually praised for producing the clearest, most coherent topics, helped by its LLM integration for natural-sounding labels. Whichever you pick, human judgement still matters for validating the topics.

The honest benefits and limitations

I am a fan, but BERTopic is not magic, and knowing where it struggles will save you a bad analysis.

On the benefits side: high topic quality from contextual embeddings, minimal preprocessing, automatic topic count, excellent handling of short text, and clear keyword-labelled topics that align with how a human would group the content. On the limitations side: it is computationally heavy and often wants a GPU; it is weak on small datasets (under about 1,000 documents, like our 100-article demo, it will still form topics but not perfectly coherent ones); it can over-produce topics (feed it 1,000 documents and you might get 200 to 500 topics unless you cap them with reduce_topics or nr_topics); it leaves some documents unassigned as outliers (that topic -1 above); and its biggest real-world catch, in my experience, is that it assigns only one topic per document, which rarely matches how content is actually written.

The good news is that two of those have practical fixes. For outliers, reduce_outliers reassigns the unplaced documents to their nearest topic, so every page gets tagged (on the demo, outliers went from 8% to 0%). For the one-topic problem, approximate_distribution slides a window across each document, scores every chunk against every topic, and returns a topic mix per document:

# reassign outliers to their nearest topic, then refresh the topic words
new_topics = topic_model.reduce_outliers(docs, topics, strategy="embeddings", embeddings=embeddings)
topic_model.update_topics(docs, topics=new_topics, vectorizer_model=vectorizer, ctfidf_model=ctfidf_model)
# a topic mix per document instead of a single topic
topic_distr, _ = topic_model.approximate_distribution(docs, window=8, stride=4, min_similarity=0.02)
topic_model.visualize_distribution(topic_distr[0])

One gotcha I hit: with reduce_frequent_words=True the similarity scores are small, and the default min_similarity=0.1 threw away almost every chunk, leaving all-zero distributions. Around 0.02 worked on my data. With that, 12% of the demo posts had a second topic above 20%. It’s an approximation, not a true mixed-membership model like LDA, but it gets you a lot closer.

That last point is exactly why I teach both approaches. BERTopic gives you crisp, coherent top-level topics; LDA, with its soft membership, is better at surfacing the multiple subtopics inside a single document. Run both and marry them and you get a far better picture for something like content categorisation than either gives alone.

Why topic models struggle on short text: 8 limitations

Short-form text is typically user-generated, defined by a lack of structure, the presence of noise and a lack of context, all of which make it difficult for machine learning models.

Topic modelling aims to identify patterns in a text corpus and extract the main themes, entities or topics, depending on what a particular model calls them. And it is notoriously more challenging when the text is shorter: when the corpus is made of tweets, say, as opposed to news articles.

Here I want to go over the main reasons topic modelling algorithms tank on short, user-generated text. I first compiled them in 2021, from the systematic literature review on topic modelling and sentiment analysis I did as part of my MSc. Each one still holds in 2026, so under each I’ve added what I do about it now.

1. There is no common definition of what short-form text is

There is no common definition of short-form text in the academic literature.

This means that when you read papers about innovations in the area, you need to know that the researchers are working with datasets with different text lengths, which makes some models look better than others.

The most common types of data used in these studies are:

• user product and service reviews
• user-generated content from social media (posts on X, formerly Twitter, Reddit posts and comments, Facebook posts, YouTube video comments, all of which vary greatly in their length limits)
• instant messages
• short message exchanges
• public forum comments
• news headlines

What I do about it now: measure your own data before you pick a model. Average words per document is a two-line calculation, and it tells you more than any benchmark. On the demo, full posts averaged 88 content words after cleaning, and subject lines averaged 3.6. Those are two completely different problems. (Platform limits keep moving, too: X’s 280-character limit now goes much higher for paying users, so “a tweet” isn’t even a stable unit of length any more.)

2. Lack of context

Short text is challenging for topic detection and sentiment extraction because it lacks contextual information, which leads to the problem of data sparsity.

As a result, general models such as bag-of-words become unsuitable for the semantic analysis of short texts, because they ignore the order and semantic relationships between words.

Even so, a review of text analysis studies in financial markets showed that the bag-of-words approach was used for feature selection in the majority of the studies reviewed. While somewhat inefficient, some approaches remain popular in the academic community.

What I do about it now: this is the problem embeddings were made for. A sentence transformer has learned from billions of sentences that “running shoes” and “trainers for jogging” mean much the same thing, so it brings the context the text itself is missing. That’s the single biggest reason I reach for BERTopic on short text.

3. The need for extensive configuration

Topic model quality depends on manipulation and refinement, which is often manual and requires time-consuming fine-tuning of model parameters. Configuration is one of the most considerable challenges in topic modelling.

Before running a topic modelling algorithm, the data should be pre-processed, and one of those steps involves removing stop words and topic-general words (TGWs). Topic-general word removal is typically done manually, which makes it challenging and time-consuming.

TGWs are problematic because they can alter the results of topic modelling: they are more likely to occur in the corpus, and so more likely to be paired with other words, which reduces the validity of the word-pair topics identified. Automation is possible, and can improve the effectiveness of the topic model.

Li, Zhang and colleagues propose the entropy weighting (EW) scheme, based on conditional entropy measured by word co-occurrences, combined with existing term weighting schemes. It automatically rewards informative words and assigns lower weights to meaningless ones, improving topic modelling performance.

What I do about it now: BERTopic has automated TGW handling built in. ClassTfidfTransformer(reduce_frequent_words=True) down-weights words that are frequent across all topics, in the same spirit as entropy weighting. On the demo titles, it turned labels like “0_for_wanted_of_to” into “0_format_file_wanted_software”. The number of topics is automatic too. What’s left to configure is mostly the HDBSCAN cluster size, and that’s one or two settings rather than a whole afternoon.

4. Bias developing in the model through human interaction

A 2019 study discusses how human interaction with topic models can be considered another research challenge.

After two experiments with non-expert users, the researchers propose that human-in-the-loop topic modelling should be developed as a form of mixed-initiative interaction, where the system and the user work collaboratively towards optimising the topic model.

What I do about it now: I treat human input as a feature, but a deliberate one. BERTopic lets you put your domain knowledge in explicitly: seed words to nudge the model towards topics you know matter (seed_topic_list), merge_topics for topics that are really one theme, and set_topic_labels to name them in your business’s language. The point is to make those choices visible and documented, rather than quietly tweaking until the output matches what you expected to see.

5. The need for extensive data pre-processing

Choosing the pre-processing technique efficiently is considered a research priority, and whole studies are devoted to it. These studies use comparative analysis to show ways of improving the effectiveness of machine learning models on tweets.

Exploring and preparing the data involves, among other things, writing functions to filter noise from the data, setting up the development environment, scaling and encoding.

This process is commonly called pre-processing, and it broadly includes three main steps: term or object standardisation, noise reduction, and word normalisation, each made up of various text analysis operations.

X (Twitter) and other social media also present the challenge of irrelevant data being collected as part of the dataset, which impacts the performance of the model. This can be resolved by adding extra steps to filter irrelevant entries out of the dataset.

What I do about it now: BERTopic needs far less pre-processing than LDA. The embedding model actually works better with natural sentences, stop words and all, so I only strip pure noise: URLs, emails, boilerplate. But the irrelevant-data point still stands. De-duplicate (in the demo, replies in a thread share a subject line, just like duplicate page titles on a site, and they formed their own topics until I removed them), and filter out entries that aren’t about your subject at all.

6. Vulnerability to overfitting

A 2018 study also argues that sentiment analysis using topic-level and word-level models is vulnerable to overfitting because of data sparsity.

This is a direct result of the data’s characteristics, namely the linguistic complexity and irregularity of user-generated short texts.

What I do about it now: the BERTopic version of overfitting is over-production: hundreds of tiny topics, each fitted to a handful of near-identical documents. Raise min_cluster_size, use reduce_topics, and always check how many documents each topic actually holds before you build anything on it.

7. Lack of consideration for abbreviations and slang

Microblogging involves flexible language, including abbreviations and slang, as opposed to structured sentences.

This is unanimously considered more challenging than traditional text for algorithmic analysis.

Part of the challenge in language interpretation is also the use of sarcasm, imagery, metaphors, similes, humour and figurative language, which rely on previous knowledge and context, as they impact the accuracy of classification of both topics and sentiment.

What I do about it now: pick an embedding model trained on text that looks like yours. General models handle common slang far better than a 2021 bag-of-words pipeline could, but niche jargon, product codes and in-jokes still trip them up. Sarcasm remains genuinely hard for every model, including LLMs, so keep a human reviewing anything that will drive decisions.

8. A lack of gold standards in academic research leads to ambiguous results

The lack of gold standards and annotated data in topic modelling and sentiment analysis reduces the academic rigour of many studies, because of subjectivity and ambiguity.

Annotation itself is time-consuming and complex, which is why the majority of studies deploy unsupervised learning algorithms.

What I do about it now: you won’t have a gold standard for your own site either, so build a cheap one. Score every model with the same coherence metric, and hand-label a sample of 50 to 100 documents to check the topics against. That’s exactly how I ran the comparison below.

Putting it to the test: BERTopic vs LDA vs NMF on titles

Here is what happened when I fitted all three models to titles only (1,373 de-duplicated subject lines from the demo, averaging 3.6 content words each), and scored each model’s top words with the same c_v coherence metric against the same reference text:

BERTopic matched NMF and clearly beat LDA, and it worked out the number of topics itself. But look at the absolute numbers: on the full posts, LDA scored around 0.66. Every model is worse on three-word documents. BERTopic doesn’t make short text easy, it makes it workable. If you can enrich short text (add the H1 and meta description to the title, or the first paragraph to the heading), do it.

If you have read my LDA guide, you will notice its titles-only scores are different (LDA 0.37, NMF 0.33). That was a separate run with different preprocessing: there the titles averaged 3.4 words after cleaning, while here I de-duplicated the subject lines first and they average 3.6 content words. On thousands of three-word documents, c_v shifts with preprocessing, the reference text and the random seed, so read both tables for the direction (every model drops sharply on short text) rather than the second decimal place.

Recognising the limitations of published research is vital for an accurate picture of what we currently know. There is a considerable number of significant studies on topic modelling and sentiment analysis of short text, but models still need refinement and optimisation to improve their accuracy and output.

Topic modelling can become a competitive advantage for businesses that want to use NLP for better predictive analytics, which is why understanding how to do it well on user-generated text is a crucial step in understanding your audience.

The same challenges show up on short-form text in SEO: titles, headings, anchor text, queries and so on. If you’d like to learn more about the tools you can use to overcome some of them, check out my guide on internal linking using machine learning.

Turning BERTopic Topics into SEO Wins

The applications mirror the LDA post, because the output serves the same goals: improve internal linking by linking articles in the same topics and subtopics; enhance it with entity extraction to enable entity-based modular linking; run competitor analysis to understand rival content libraries fast; and do market analysis by clustering YouTube or Reddit content to read a competitive landscape quickly. My standing challenge: on your next content audit, run BERTopic, LDA and Google’s text classification on the same set and see which gives the most useful labels.

Where to Take BERTopic Next

Read the companion post on topic modelling with LDA to see the classic approach BERTopic builds on, and what clustering is and where it fits in SEO for the wider context. For hands-on keyword work, our KeyBERT keyword clustering guide is a great next step, and the free supervised vs unsupervised (BERTopic) keyword-to-topic mapping notebook shows BERTopic on keywords, the shortest text of all.

Related glossary terms

• BERTopic
• BERT
• HDBSCAN
• Dimensionality reduction
• Topic modeling
• c-TF-IDF
• UMAP
• Embedding
• Stop words
• Coherence score