Corpus (Text Corpus)

A corpus (plural: corpora) is the collection of text you analyse or train a model on. It might be every page on your site, a set of customer reviews, a keyword list, a batch of video transcripts or the huge body of web text behind a large language model. Each item inside the corpus is usually called a document, and many NLP methods only make sense relative to the corpus they run on, so what you put in shapes what comes out.

How it works

Working with a corpus usually follows the same steps, whatever the method:

• Decide what counts as a document. A whole page, a paragraph, a title, a single review or a tweet are all valid choices, and the choice changes the results. A content audit normally treats each URL as one document.
• Clean and prepare the text. Tokenization splits each document into tokens, and steps such as removing stop words and lemmatisation reduce noise. The set of unique tokens across the whole corpus becomes its vocabulary.
• Calculate corpus-level statistics. TF-IDF weights a word as more important to a document if it is common there but rare across the rest of the corpus. Topic modelling methods such as LDA look for patterns in how words co-occur across all the documents in the corpus.

Because these scores depend on the whole collection, adding or removing documents changes them. The same page can get different TF-IDF weights or a different topic assignment in a different corpus.

Some corpora are built and curated for research. NLTK, the Python NLP library, ships with reference corpora such as the Brown Corpus, which its documentation describes as the first million-word electronic corpus of English, created in 1961 at Brown University.

How it’s used in SEO and marketing

• Topic modelling a content audit: the corpus is your site’s pages, analysed as one batch to find the main themes and how pages group together.
• Keyword research: MLforSEO’s EAV framework starts by building a keyword corpus from several sources (Search Console, keyword tools, autocomplete, competitors, internal site search) and then runs entity extraction across it.
• Text classification: the Google Natural Language API can classify any text corpus you give it, from web pages to video transcripts to tweets.
• Fuzzy and TF-IDF matching: matching titles or meta descriptions against a keyphrase list weighs each token by how rare it is in the corpus, which makes matches context-sensitive.
• Review analysis: a corpus of customer reviews feeds sentiment and entity analysis.

Example

Say you run LDA on 200 blog posts. Most of them mention “machine learning”, so across this corpus it is a weak signal, while “fuzzy matching” appears in only a handful of posts and stands out strongly in those. Now add 500 product pages to the same corpus. The vocabulary, the word weights and the topics all shift, even though the blog posts themselves did not change. That is why you define the corpus deliberately before you run anything.

Limitations and things to watch

• Boilerplate pollutes the corpus. Navigation, footers and cookie banners repeat on every page and can dominate the results, as MLforSEO shows in why navigation elements outrank your brand in entity analysis.
• Short documents are harder. Topic models struggle more when the corpus is made of tweets or keywords than when it is made of long articles.
• A corpus reflects what is in it. Pre-trained models learn from their own training corpus, so niche terms that were rare there tend to be handled poorly.
• Duplicates and mixed languages skew counts, so de-duplicate and split by language first.

Learn it on MLforSEO

• Topic Modelling for SEO with LDA (Latent Dirichlet Allocation)
• Topic Modelling for SEO with BERTopic
• What Fuzzy Matching Is, and Where It Fits in SEO
• Web Content Clustering: Topic Modeling with LDA (Notebook)
• Introduction to Machine Learning for SEO (course)