Topic Modelling for SEO with LDA (Latent Dirichlet Allocation)

Before there were transformer embeddings and AI Overviews, there was a quietly revolutionary idea: that a document is not one topic but a mixture of many, and that a machine could work out those topics from nothing but the words. That idea is Latent Dirichlet Allocation, or LDA, and understanding it tells you something important about how modern search learned to think in themes rather than links.

Topic modelling is also one of the natural language processing techniques I find most needed in my own work. These days I often see it lumped together with entity extraction, but the aim of it remains the same: pattern recognition in a corpus of text.

This post explains what LDA is, how it works, why it emerged, and how to actually run it on your own content, both with no code and in Python. It is also one of the hands-on practicals in my Introduction to Machine Learning for SEO course, so I will point you to the templates and the course for the full step-by-step. In this guide you will find:

• What topic modelling is, and the main families of approaches
• What LDA is, how it works, and why it exists
• What LDA is good for, and what it is criticised for
• Hybrid LDA methods and non-LDA alternatives, and where they fall short
• How to run LDA on your content: a no-code route (with a video walkthrough) and refreshed Python code
• How to judge your topics, and how to turn them into SEO wins like internal linking

All of the code in this guide is in one runnable notebook: grab it from the MLforSEO code hub, open it in Google Colab and run it top to bottom, on the demo data or on your own crawl.

What topic modelling is: an overview

Topic modelling is a text processing technique aimed at overcoming information overload by seeking out and demonstrating patterns in textual data, identified as the topics. It enables an improved user experience, allowing analysts to navigate quickly through a corpus of text or a collection, guided by the identified topics.

Topic modelling is typically performed via unsupervised learning, with the output of running the models being a summary overview of the discovered themes. In other words, you don’t need labelled data, and you don’t tell the model what the topics are. It tells you.

Detecting topics can be done in both online and offline modes. When done online, it aims to discover dynamic topics over time as they appear (think of emerging conversations on social media). When done offline, it is retrospective, considering the documents in the corpus as a batch. A content audit of your site is an offline job.

There are four main approaches to topic detection and modelling:

• keyword-based approaches
• probabilistic topic modelling (this is where LDA sits)
• aging theory (modelling how topics emerge, grow and fade over time)
• graph-based approaches

Approaches can also be categorised by the techniques used for topic identification, which gives three groups:

• clustering
• classification
• probabilistic techniques

Keep those families in mind, because the history of topic modelling is largely the story of LDA (probabilistic) being challenged by clustering on top of embeddings, which is exactly what BERTopic does.

What LDA is

LDA is a generative probabilistic model used in natural language processing to uncover hidden topics in a collection of documents by spotting the patterns in how words co-occur. You feed it a corpus, your site content, a competitor’s, a set of YouTube transcripts, and it works backwards to tell you what the topics are, which words define each one, and how strongly each document belongs to each topic.

I want to be upfront: LDA is old, introduced by David Blei, Andrew Ng and Michael Jordan in their 2003 paper “Latent Dirichlet Allocation” in the Journal of Machine Learning Research, where they framed it as a three-level hierarchical Bayesian model for collections of text. It is still one of the most classical ways to do topic modelling, and many modern approaches are hybrids built on top of it, so it is genuinely worth understanding. We will compare it head to head with the more modern BERTopic in a companion post.

How LDA works

LDA imagines that every document is a mix of subtopics, and every topic is a bucket of words that tend to appear together.

The model assumes text is generated like this: a document is, say, 85% politics, 10% finance, 5% history, and for each word the machine picks one of those topics and emits a likely word from it. Crucially, LDA does not actually write documents, it assumes this is how they were written, then reverse-engineers the process to recover the dominant topics across your whole collection. The output is a list of topics, the terms that define each, and a probability score for how strongly each document belongs to each topic.

Put another way, you have different words that can stand as descriptors for a particular topic, and the algorithm calculates the probability of each word in a document aligning to one of the core topics it identifies. What this allows is someone like us coming in, looking at the words that describe a particular subtopic, and giving them names. What do these terms represent? What is the name of this topic? And then seeing how often each of these subtopics is represented in a given document. Topic modelling, in a nutshell, is pattern recognition in large text-based corpora of data.

In the vocabulary of machine learning clustering, LDA is a Bayesian, conditional-probabilistic model; it is a soft (fuzzy) clustering approach, so documents and words can belong to multiple topics with varying probability; and it is unsupervised, so no labels are required.

Under the hood: the technical definition

For the more technically curious: LDA is a Bayesian hierarchical probabilistic generative model for collections of discrete data. It operates on an exchangeability assumption for the words and topics in a document (the order of words doesn’t matter, which is why it is called a bag-of-words model). It models documents as discrete distributions over topics, and topics in turn as discrete distributions over the terms in the documents.

The original LDA method uses a variational expectation maximisation (VEM) algorithm to infer the topics. Later on, stochastic sampling inference based on Gibbs sampling was introduced, which improved performance in experiments and has since been used frequently. (If you use gensim, as in the code below, it trains LDA with online variational Bayes, a scalable descendant of the original approach. If you specifically want Gibbs sampling, the tomotopy library has it.)

Blei and his colleagues, who first introduced LDA, demonstrated its superiority against the probabilistic LSI model. LSI (Latent Semantic Indexing) uses linear algebra and bag-of-words representations to extract words with similar meanings (you will also see it called LSA).

Why LDA even exists (and why SEOs should care)

LDA emerged to solve a problem that will sound familiar. As information got digitised, organising documents purely by links and keyword matching proved too limiting, a page with lots of links that loosely matches a query is not necessarily the best answer, especially for niche queries. Researchers wanted systems that organised documents by themes and subtopics, and that could zoom into the niche interests buried inside a page.

David Blei, one of LDA’s authors, describes the motivation as three problems, each with a solution:

1. Knowledge was exploding and getting harder to search. As our collective knowledge grows, discovering information gets more and more difficult. The solution: computational tools that help organise, search and understand vast amounts of information.
2. Query-and-link systems were good, but limiting. Ranking documents purely by links and query matching was too crude, because a heavily linked page that only loosely matches a query is not necessarily the best answer, especially for niche searches. The solution: systems that organise documents by themes, in other words topics and subtopics.
3. There was no way to zoom in and out of topics. Content was organised by links, not by the themes different readers actually cared about, and there was no way to highlight patterns within a topic. The solution: topic models that assume a document has multiple subtopics within it.

LDA’s answer was to assume every document contains multiple sub-topics, each of potential interest to different people, and to surface them computationally. And this is how LDA was born. (It wasn’t the very first topic model, LSI dates from 1990 and probabilistic LSI from 1999, but it was the first fully generative probabilistic one, and it is the model most people mean when they say “topic modelling”.) That shift, from links to topics and term understanding, is a direct ancestor of how modern search, and now AI search, decides what a page is really about. Knowing LDA is knowing where topical relevance came from.

What is LDA good for?

In my literature review, LDA was the technique most commonly used as part of models across all the studies I reviewed. Here is where the research says it earns its keep.

1. Strategic business optimisation

LDA is considered of value for strategic business optimisation: getting a structured overview of large volumes of text quickly enough to act on it.

Here I’d like to specifically mention the role and huge impact LDA can have for SEOs. It can help you understand large website structures, topics and subtopics quickly, which helps you establish a basic structure for internal link efforts and internal link optimisation. That in turn can help increase a website’s visibility in search engines and its organic performance.

2. Competitive advantage through a better understanding of user-generated text

A 2018 study demonstrates the value of LDA as a way of improving a company’s competitive advantage by extracting information from users’ online reviews, then classifying the topics by sentiment.

3. A better understanding of your users

LDA-based topic modelling has also been used to characterise the personality traits of users, based on the text they publish online.

In my own study, I used LDA topic modelling to categorise users into stages of their customer journey, based on the short, user-generated text they posted on social media about a product or company.

4. Understanding customer complaints and making customer service more efficient

In a 2019 study, LDA topic modelling was used to analyse consumer complaints submitted to the US Consumer Financial Protection Bureau. Predetermined labels were used for classification, which improved the efficiency of the complaint-handling department through task automation.

What is LDA criticised for?

LDA is a classic for a reason, but the literature is just as clear about its weak spots. Knowing them will save you from a bad analysis.

1. Inability to scale

LDA has been criticised for not scaling well, because of the linearity of the technique it is based on. Other variants, such as pLSI (the probabilistic variant of LSI), address this challenge by using a statistical foundation and working with a generative data model.

2026 note: in practice, scale is much less of a blocker than it was. Online variational Bayes (what gensim uses) streams documents in chunks, and gensim’s LdaMulticore spreads training across CPU cores, so tens of thousands of pages run comfortably on a laptop or in Colab.

2. Assuming document exchangeability

Although efficient and frequently used, LDA is criticised for its assumption of document exchangeability. This can be restrictive in contexts where topics evolve over time, such as news or social media. (Dynamic topic models, available in gensim as LdaSeqModel, and BERTopic’s topics_over_time() were built for exactly this.)

3. Commonly neglecting co-occurrence relations

LDA-based models are criticised for commonly neglecting co-occurrence relations across the documents analysed. This results in incomplete information, and an inability to discover latent co-occurrence relations through context or other bridge terms.

Why is this important? It can prevent topics that are important but rare from being detected.

This criticism is also shared in a later study, where the authors propose a model specifically tailored for topic modelling on online social networks. They demonstrate that even shallow machine learning clustering techniques, applied to neural embedding representations, perform more efficiently than LDA. Sound familiar? That is the core idea behind BERTopic.

4. Unsuitable for short, user-generated text

Short-form text is typically user-generated, defined by a lack of structure, the presence of noise and a lack of context, which makes it difficult for machine learning models. Hajjem and Latiri (2017) criticise the LDA approach as unsuitable for short-form text, and propose a hybrid model that uses mechanisms typical of the field of information retrieval.

LDA-hybrid approaches have also been proposed to address these limitations. However, even they perform sub-optimally on short-form text, which calls into question the efficiency of LDA on noisy, unstructured social media data.

This one matters for SEOs more than you might think, because a lot of our data is short: titles, headings, anchor text, queries, reviews. I ran the experiment in the notebook to show you. Same documents, same pipeline, same eight topics, once on the full text and once on the titles only (an average of 3.4 words each after cleaning):

Coherence roughly halves, and the title-only topics are a soup (“hockey, help, questions, april, needed, nasa”). A three-word title just doesn’t give LDA enough co-occurrence evidence to work with. I go through the eight reasons topic models tank on short text, and what to do about each, in the BERTopic guide. That guide re-runs the titles-only test with de-duplicated titles and gets slightly different numbers (LDA 0.33, NMF 0.41), a reminder that c_v on very short text moves with preprocessing, so the drop matters more than the exact decimals.

Hybrid LDA methods

To address some of these limitations, models that learn vector representations of words were introduced. By learning vector representations of words and hidden topics, they are shown to have a more effective classification performance on short-form text.

Yu and Qiu propose a hybrid model where the user-LDA topic model is extended with the Dirichlet multinomial mixture and a word vector tool. It performed best when compared with other hybrid models, or with LDA alone, on microblog (in other words, short) text.

Another conceptually similar approach can be applied to Twitter data: hierarchical latent Dirichlet allocation (hLDA), which aims to automatically mine the hierarchical dimension of tweets’ topics by using word2vec (a vector representation technique). By doing so, it extracts semantic relationships between words in the data to obtain a more effective dimension.

Other approaches, such as the non-negative matrix factorisation (NMF) model, have also been found to perform better than LDA on short text under similar configurations. (In my 2026 notebook run, NMF beat LDA on full text in some runs and lost on titles in others, so test both on your data rather than taking any one study’s word for it.)

The direction of travel since 2021 has been clear: from word vectors bolted onto LDA, to models that start from contextual sentence embeddings, such as BERTopic and Top2Vec. That’s why the companion post exists.

LDA alternatives

Apart from LDA, there are numerous other developments in the field of topic discovery. Considering how little academic attention they have received, they appear to have critical limitations that remain unaddressed.

In a 2017 study, a hierarchical approach to topic detection is proposed, where words are treated as binary variables and allowed to appear in only one branch of the hierarchy. Although efficient compared with LDA, this approach is unsuitable for user-generated content, or social media short text, because of the language ambiguity that characterises that data.

A Gaussian Mixture Model can also be used for topic modelling of news articles. This model represents text as a probability distribution as a way of discovering topics. Although better than LDA, it will again likely perform less coherently on user-generated short texts, because of their lack of structure and data sparsity.

Another model based on Formal Concept Analysis (FCA) was proposed for topic modelling of Twitter data. It can detect new topics based on information from previous topics. Yet it fails to generalise well, meaning it is unreliable and sensitive to topics it has not been trained on.

Other models, such as TG-MDP (topic-graph Markov decision process), consider the semantic characteristics of textual data and automatically select an optimal set of topics with low time complexity. This approach suits only offline topic detection. Even so, it offers promising results when benchmarked against LDA-based algorithms (GAC, LDA-GS, KG).

What can these models be used for, you might be wondering?

• detecting emerging topics in microblogging communities like X (Twitter), Reddit and other sites
• observing how topics evolve over a period of time

How to run LDA on your content

There are two routes, and the course walks through both end to end.

The no-code route

The no-code route uses a free web app (built by Cornell). You crawl and export your content, upload it with a stopword list, choose how many topics and how many iterations to run, then explore and download the output. Start to finish it takes about 30 minutes, and there is a Google Sheets template to organise the results into topic-to-topic similarity and per-page topic probabilities.

I recorded a step-by-step video tutorial on how to do topic modelling with jsLDA, the no-code, publicly available web app that runs LDA, originally developed at Cornell by David Mimno.

In a nutshell, the video shows you how to crawl and export the content on your website, upload your files to the web app, fine-tune the model, then download the files, explore them and build them into your internal linking audit deliverable.

In the end, you’re going to have a couple of files. One shows topic-to-topic similarity: how often the different topics on your website co-occur, or whether some of them don’t co-occur at all and maybe shouldn’t be related. The other shows which topic each piece of content on your site, each page, belongs to.

Get your copy of the Google Sheets template for organising the topic modelling exports

Why should you do this?

• You will get a baseline overview of the website (or any other text-based corpus) in less than 30 minutes, regardless of its size.
• You will save yourself a ton of time.
• You will test something new, risk-free, and if you are a newbie in the machine learning world, code-free, too!

The Python route

The Python route in Google Colab does the same thing but scales, and adds things the web app cannot: a word cloud of your corpus, top keywords per topic, an interactive pyLDAvis visualisation, and proper quality evaluation. This is where topic modelling stops being a black box.

Here are the essentials, tested in October 2026 with gensim 4.4. The full notebook loads your crawl export (one row per page, with the URL and the main body text) or a demo dataset, and adds the tuning, charts and exports. First, clean and tokenise. LDA is a bag-of-words model, so pre-processing does a lot of the heavy lifting:

# !pip install -q "gensim>=4.4" "pyLDAvis>=3.4" wordcloud
import re
from gensim.corpora import Dictionary
from gensim.models import LdaModel, Phrases, CoherenceModel
from gensim.models.phrases import ENGLISH_CONNECTOR_WORDS
from gensim.parsing.preprocessing import STOPWORDS
from gensim.utils import simple_preprocess
# add your brand name, boilerplate and site-wide words
STOP = STOPWORDS.union({"cookie", "menu", "yourbrand"})
def tokenise(text):
    text = re.sub(r"\S+@\S+|https?://\S+", " ", str(text))   # strip emails and URLs
    return [t for t in simple_preprocess(text, deacc=True, min_len=3) if t not in STOP]
tokens = [tokenise(t) for t in df["text"]]          # df: one row per page
bigram = Phrases(tokens, min_count=5, threshold=20, connector_words=ENGLISH_CONNECTOR_WORDS)
tokens = [bigram[doc] for doc in tokens]            # "internal_linking" stays one token
dictionary = Dictionary(tokens)
dictionary.filter_extremes(no_below=5, no_above=0.5)   # drop rare words and topic-general words
corpus = [dictionary.doc2bow(doc) for doc in tokens]

That no_above=0.5 line is doing something important: it drops any word that appears in more than half of your pages. Words like that (your brand name, “services”, “contact”) carry no topical signal and blur topics together. Then train the model:

lda = LdaModel(corpus=corpus, id2word=dictionary, num_topics=8,
               alpha="auto", eta="auto",          # learn the priors from the data
               passes=10, iterations=200, random_state=42)
for t in range(8):
    print(t, ", ".join(w for w, _ in lda.show_topic(t, topn=10)))

Then the visualisation:

import pyLDAvis, pyLDAvis.gensim_models
vis = pyLDAvis.gensim_models.prepare(lda, corpus, dictionary, mds="mmds")
pyLDAvis.save_html(vis, "lda_pyldavis.html")       # in Colab: pyLDAvis.enable_notebook(); vis

Two things trip people up here if they follow older tutorials: the module is now pyLDAvis.gensim_models (it was pyLDAvis.gensim), and on current NumPy and SciPy the default layout can crash with a “complex128 is not JSON serializable” error, which mds="mmds" avoids.

The pyLDAvis visualisation is the one worth lingering on, because it makes an abstract model tangible. On the left it plots your topics as bubbles, sized by how much of the whole corpus each represents and positioned by how similar they are to one another; click a topic and the right-hand panel shows its 30 most relevant terms. Suddenly you can see that, say, topic two accounts for 10% of all the words in a competitor’s content and is clearly about a subject you have never covered, an instant, visual content-gap finding. The notebook also gives you a word cloud of the cleaned corpus for a top-level sense of the whole dataset, the top keywords per topic as plain lists, and a chart of the top articles per topic ranked by their probability score, which is exactly the input you need for a topic-based internal-linking exercise.

Prefer to read someone else’s walkthrough first? These tutorials are a good way to kick off a topic modelling journey in Python:

• Topic Modeling in Python: Latent Dirichlet Allocation (LDA) by Shashank Kapadia
• Topic Modelling in Python with NLTK and Gensim by Susan Li
• Latent Semantic Analysis using Python by Avinash Navlani (LSA rather than LDA, a useful comparison)
• Linear Discriminant Analysis With Python by Jason Brownlee. A word of warning: this is the other LDA, a supervised classification method, not a topic model. I am keeping it in the list because the acronym clash catches a lot of people out, so make sure you know which LDA a tutorial means before you follow it.

Some of these were written for gensim 3 and older pyLDAvis versions, so expect a few renamed imports. The MLforSEO notebook is up to date.

Judging whether your topics are any good

Topic modelling is exploratory, not one-and-done, so evaluating quality matters. LDA uses two metrics.

Perplexity measures how well the model predicts unseen documents, and lower is better. Coherence measures how semantically related the words within a topic are, how much sense they make to a human, and higher is better. If your scores disappoint, you tune the hyperparameters: the number of topics, plus alpha (the document-to-topic distribution) and beta (the topic-to-word distribution). Too high an alpha crams too many topics into each document and blurs the topics; too high a beta makes topics too broad. You test configurations, find the highest coherence, and rebuild with those settings.

cm = CoherenceModel(model=lda, texts=tokens, dictionary=dictionary, coherence="c_v")
print("Coherence (c_v):", round(cm.get_coherence(), 3))       # higher is better, 0 to 1
bound = lda.log_perplexity(holdout_corpus)                    # held-out pages
print("Perplexity:", round(2 ** (-bound)))                    # lower is better

A gotcha worth knowing: gensim’s log_perplexity() doesn’t return perplexity. It returns a negative per-word bound, and perplexity is 2 ** (-bound). Plenty of tutorials compare the raw bounds and pick the wrong model. And when the two metrics disagree, as they often do, trust coherence: research by Chang and colleagues (2009) found that models with better perplexity often produce topics humans find less meaningful. On the demo data, the notebook’s tuning grid picked 8 topics with an asymmetric alpha, at a coherence of about 0.68.

Last of all, look at the topics yourself. In the demo, one topic was made of nothing but a prolific poster’s email signature, a nice reminder that a model will happily find a “topic” in your boilerplate if you don’t clean it out first.

Turning LDA Topics into SEO Wins

The point of all this is action. The strongest use is internal linking: combine the per-document topic distribution with the topic-to-topic correlations to find articles on co-occurring subtopics, then link them, individually or through modular “related content” blocks. Pair it with entity extraction and you get entity-based modular linking. The same analysis powers competitor analysis (what is their content library actually about) and market analysis (map the YouTube or Reddit landscape by swapping in transcripts or posts as your corpus).

Here is the internal-linking step in code. Because LDA gives every page a probability for every topic, you can compare two pages’ topic mixes directly. The notebook uses Hellinger distance, the standard way to compare probability distributions, and returns the five closest pages for every URL:

import numpy as np
# every page's probability for every topic (soft membership)
doc_topic = np.array([[p for _, p in lda.get_document_topics(bow, minimum_probability=0)] for bow in corpus])
S = np.sqrt(doc_topic)
H = np.sqrt(np.clip(1 - S @ S.T, 0, None))          # Hellinger distance between every pair of pages
np.fill_diagonal(H, np.inf)
link_candidates = {df.url[i]: [df.url[j] for j in np.argsort(H[i])[:5]] for i in range(len(df))}

Treat the output as a shortlist, not a decision: a human still checks that each link genuinely helps the reader and that the anchor text makes sense. For the full audit workflow, see my guide to internal link audits and topic clustering with machine learning.

Where to Take LDA Topic Modelling Next

To recap: although there are many approaches to topic modelling, LDA has become the most commonly used. Because of its limitations, many hybrid approaches have been developed to improve topic accuracy and relevance, and these often challenge LDA’s probabilistic hierarchical structure. Non-LDA approaches have also been developed, but they are not well suited to short-form text analytics.

LDA is the classic, but it is not the only option, and often not the best one for short text. Read the companion post on topic modelling with BERTopic to see the modern, embedding-based approach, and the eight reasons topic models struggle on short text, then decide which fits your data, or, as I often suggest, run both and marry the results. If you are new to the wider technique, start with what clustering is and where it fits in SEO, and if your corpus is keywords rather than pages, see the different ways to map keywords to topics.

Related glossary terms

• LDA (Latent Dirichlet Allocation)
• Topic modeling
• Soft (fuzzy) clustering
• Hyperparameter tuning
• Clustering
• Coherence score
• NMF (Non-negative Matrix Factorization)
• LSA (Latent Semantic Analysis)
• Bag of words