Lexical or morphological analysis

Lexical or morphological analysis is the first phase of natural language processing. It breaks text into words or tokens and analyses their internal structure (roots, prefixes, suffixes, tense and number) so that later phases work with consistent units. It is what lets a system treat “running shoes” and “running shoe” as the same thing.

How it works

NLP is often described in five phases: lexical or morphological analysis, syntax analysis, semantic analysis, discourse integration and pragmatic analysis. The idea borrows from compiler design, where a lexer first turns source code into tokens before the code is parsed and interpreted. In language, the lexical phase typically covers:

• Tokenisation: splitting text into words, punctuation and numbers.
• Normalisation: lowercasing, handling accents and standardising spellings.
• Stemming: chopping words to a crude stem with rules, so “studies” becomes “studi”.
• Lemmatisation: mapping each word to its dictionary form using vocabulary and part of speech, so “studies” becomes “study” and “better” becomes “good”.
• Morphological features: tagging each token with tense, number, person and similar properties.

Libraries such as spaCy and NLTK do all of this in a few lines; spaCy exposes a lemma and morphological features for every token.

How it’s used in SEO and marketing

• Normalising and de-duplicating keyword lists before clustering or counting.
• Cleaner n-gram and word-frequency analysis of queries, titles and reviews.
• Preparing text for topic modelling and classic machine learning, alongside stop word removal.
• Multilingual keyword research, where languages with rich morphology produce many forms of the same word.

Example

Take three queries: “best running shoe”, “best running shoes” and “best shoes for running”. Lemmatising turns “shoes” into “shoe”, so the first two collapse into one pattern in your counts. The third still differs in word order, which is a job for syntax or semantic analysis, or for embeddings, rather than the lexical phase.

Limitations and things to watch

• Stemmers over- and under-cut words (“university” and “universe” can share a stem).
• Rules and dictionaries are language-specific, so check support for each market.
• Modern transformer models use sub-word tokenisation and do not need explicit lemmatisation, but it remains useful for counting and rule-based work.

Learn it on MLforSEO

• How to do Syntax Analysis with Google’s Natural Language API in Google Sheets
• What Entity Extraction Is, and Where It Fits in SEO
• What Machine Learning Clustering Is, and Where It Fits in SEO
• Introduction to Machine Learning for SEO (course)