Keyword Extraction

Keyword extraction is a natural language processing task that identifies the words and phrases that best represent what a text is about. Given a page, a review or a list of queries, it returns a short list of key terms. In SEO it is the automated version of reading a document and noting its main themes, and it underpins tasks from content tagging to keyword clustering.

How it works

There are three broad families of methods:

• Statistical: score terms by how frequent they are in one document and how rare across a collection, as in TF-IDF, or by position and co-occurrence, as in RAKE and YAKE.
• Linguistic and API-based: use part-of-speech patterns to find noun phrases. Amazon Comprehend’s key phrases feature, for example, returns noun phrases with a confidence score.
• Embedding-based: KeyBERT embeds the document and candidate n-grams with a BERT-based model, then ranks candidates by cosine similarity to the document. Options such as Maximal Marginal Relevance make the list more diverse.

Keyword extraction differs from entity extraction: entities are named things (brands, people, places) linked to a knowledge base, while extracted keywords can be any important phrase, such as “redirect mapping” or “battery life”.

How it’s used in SEO and marketing

• Keyword research: extracting the core term from long-tail queries, then grouping queries by it.
• Content audits: summarising what each page is about, at scale, to spot overlaps and gaps.
• Competitor analysis: comparing the key phrases on your pages with those on ranking competitors.
• Tagging and internal linking: suggesting tags or anchor candidates.
• Voice of customer: pulling recurring phrases from reviews and support tickets.

Example

In MLforSEO’s KeyBERT guide, a list of keywords is passed through KeyBERT, which returns the one or two words that most strongly represent each keyword semantically. Keywords that share an extracted term are grouped, giving a fast first draft of a keyword clustering that you can visualise in Looker Studio and refine by hand.

Limitations and things to watch

• Extracted terms are only as good as the input; boilerplate and navigation text need removing first.
• Statistical methods miss synonyms; embedding methods can return near-duplicate phrases unless you add diversity.
• Very short texts, such as single queries, give the method little to work with.
• Results are candidates. A person still decides which terms are meaningful for the business.

Learn it on MLforSEO

• How to do keyword clustering with KeyBERT
• KeyBERT for SEOs: Google Colab Template
• Keyword Cluster (KeyBERT) Looker Studio Demo Visualization Dashboard for SEO
• Comparing NLP APIs for Entity and Sentiment Analysis (vs Generative AI)
• Introduction to Machine Learning for SEO (course)