Lemmatisation and Stemming

Lemmatisation and stemming are text preprocessing steps that reduce different forms of a word to a shared base, so “running”, “runs” and “run” can be counted as the same word. Stemming does this by cutting off word endings with rules. Lemmatisation uses vocabulary and grammar to return the proper dictionary form, called the lemma.

How it works

Stemming applies suffix-stripping rules. The classic Porter stemmer (1980) and its successor, Snowball, turn “studies” into “studi” and “marketing” into “market”. Stems need not be real words, and the rules know nothing about meaning, so irregular forms such as “ran” are left alone.

Lemmatisation looks words up in a lexicon and usually considers part of speech, which comes from syntax analysis. It maps “ran” to “run” and “studies” to “study”, and can keep “meeting” as a noun or reduce it to “meet” as a verb. It is slower but more accurate. spaCy and NLTK offer both approaches, and the Google Cloud Natural Language API returns a lemma for every token in its syntax analysis.

Both usually follow tokenisation and sit alongside stop word removal in a preprocessing pipeline.

How it’s used in SEO and marketing

• Grouping keyword variants before n-gram analysis, so “audit”, “audits” and “auditing” add up.
• Preparing text for TF-IDF, LDA and other bag-of-words models.
• Cleaner keyword extraction from page content or reviews.
• Understanding search engines: query expansion and synthetic query generation include lexical steps such as stemming, lemmatisation and spelling correction.

Example

In MLforSEO’s syntax analysis template, each token comes back with its lemma, the canonical form useful for text normalisation and keyword extraction. Running it over a set of product reviews lets you count “delivered”, “delivering” and “deliveries” together under “deliver” and “delivery” when you summarise what customers mention most.

Limitations and things to watch

• Stemming can merge words with different meanings (“marketing” and “market”) and split ones that belong together.
• Results vary by language. Morphologically rich languages need language-specific tools.
• Modern embedding and transformer models use subword tokenisation and learn from word forms directly, so they usually do not need stemming or lemmatisation, and applying it can remove useful signal.
• Keep the original text alongside the processed version so you can report real phrases.

Learn it on MLforSEO

• How to do Syntax Analysis with Google’s Natural Language API in Google Sheets
• What Machine Learning Clustering Is, and Where It Fits in SEO
• Google Sheets Syntax Analysis Template (resource).
• Semantic ML-enabled Keyword Research course, lesson on synthetic queries.