Topic Modelling for SEO with LDA (Latent Dirichlet Allocation)

Before there were transformer embeddings and AI Overviews, there was a quietly revolutionary idea: that a document is not one topic but a mixture of many, and that a machine could work out those topics from nothing but the words. That idea is Latent Dirichlet Allocation, or LDA, and understanding it tells you something important about how modern search learned to think in themes rather than links.

This post explains what LDA is, how it works, why it emerged, and how to actually run it on your own content, both with no code and in Python. It is a practical from the clustering module, so I will point you to the templates and the course for the full step-by-step.

What LDA is

LDA is a generative probabilistic model used in natural language processing to uncover hidden topics in a collection of documents by spotting the patterns in how words co-occur. You feed it a corpus, your site content, a competitor’s, a set of YouTube transcripts, and it works backwards to tell you what the topics are, which words define each one, and how strongly each document belongs to each topic.

I want to be upfront: LDA is old, dating to the early 2000s. It is still one of the most classical ways to do topic modelling, and many modern approaches are hybrids built on top of it, so it is genuinely worth understanding. We will compare it head to head with the more modern BERTopic in a companion post.

How LDA works

LDA imagines that every document is a mix of subtopics, and every topic is a bucket of words that tend to appear together.

The model assumes text is generated like this: a document is, say, 85% politics, 10% finance, 5% history, and for each word the machine picks one of those topics and emits a likely word from it. Crucially, LDA does not actually write documents, it assumes this is how they were written, then reverse-engineers the process to recover the dominant topics across your whole collection. The output is a list of topics, the terms that define each, and a probability score for how strongly each document belongs to each topic.

In the vocabulary of the previous lessons, LDA is a Bayesian, conditional-probabilistic model; it is a soft (fuzzy) clustering approach, so documents and words can belong to multiple topics with varying probability; and it is unsupervised, so no labels are required.

Why LDA even exists (and why SEOs should care)

LDA emerged to solve a problem that will sound familiar. As information got digitised, organising documents purely by links and keyword matching proved too limiting, a page with lots of links that loosely matches a query is not necessarily the best answer, especially for niche queries. Researchers wanted systems that organised documents by themes and subtopics, and that could zoom into the niche interests buried inside a page. That shift, from links to topics and term understanding, is a direct ancestor of how modern search, and now AI search, decides what a page is really about. Knowing LDA is knowing where topical relevance came from.

How to run LDA on your content

There are two routes, and the course walks through both end to end.

The no-code route uses a free web app (built by Cornell). You crawl and export your content, upload it with a stopword list, choose how many topics and how many iterations to run, then explore and download the output. Start to finish it takes about 30 minutes, and there is a Google Sheets template to organise the results into topic-to-topic similarity and per-page topic probabilities. You can get started with this simple tutorial on YouTube

I highly recommend working on this in Python, though. The Python route in Google Colab does the same thing but scales, and adds things the web app cannot: a word cloud of your corpus, top keywords per topic, an interactive pyLDAvis visualisation, and proper quality evaluation. This is where topic modelling stops being a black box.

Judging whether your topics are any good

Topic modelling is exploratory, not one-and-done, so evaluating quality matters. LDA uses two metrics.

Here’s what these two scores mean:

• Perplexity measures how well the model predicts unseen documents, and lower is better.
• Coherence measures how semantically related the words within a topic are, how much sense they make to a human, and higher is better.

If your scores disappoint, you tune the hyperparameters: the number of topics, plus alpha (the document-to-topic distribution) and beta (the topic-to-word distribution). Too high an alpha crams too many words into documents and blurs the topics; too high a beta makes topics too broad. You test configurations, find the highest coherence, and rebuild with those settings.

Turning topics into SEO wins

The point of all this is action. The strongest use is internal linking: combine the per-document topic distribution with the topic-to-topic correlations to find articles on co-occurring subtopics, then link them, individually or through modular “related content” blocks. Pair it with entity extraction and you get entity-based modular linking. The same analysis powers competitor analysis (what is their content library actually about) and market analysis (map the YouTube or Reddit landscape by swapping in transcripts or posts as your corpus).

Where this goes next

LDA is the classic, but it is not the only option, and often not the best one for short text. Read the companion post on topic modelling with BERTopic to see the modern, embedding-based approach, then decide which fits your data, or, as I often suggest, run both and marry the results. If you are new to the wider technique, start with what clustering is and where it fits in SEO.

Related glossary terms

• LDA (Latent Dirichlet Allocation)
• Topic modeling
• Soft (fuzzy) clustering
• Hyperparameter tuning
• Clustering