NLP (Concepts & Pipeline)
Stop Words
In shortVery common words such as "the", "and" and "of" that are often removed before text analysis so the meaningful terms stand out.
Stop words are very common words, such as articles, prepositions and pronouns (“the”, “a”, “of”, “is”), that carry little topical meaning on their own. In many text analysis tasks they are removed after tokenisation so that counts, topics and clusters focus on the meaningful terms. Stop word lists are language-specific and often customised for a project.
How it works
- Libraries such as NLTK, spaCy and scikit-learn include default stop word lists in many languages.
- After splitting text into tokens, any token on the list is dropped.
- Custom lists add domain words that appear everywhere in your dataset (your brand name, “seo” on an SEO blog, “product” in reviews), sometimes called topic-general words.
- Weighting schemes such as TF-IDF reduce the influence of common words automatically, which is why some workflows remove fewer stop words.
How it’s used in SEO and marketing
- Topic modelling: LDA needs stop words and topic-general words removed, or topics fill up with filler.
- N-gram analysis of titles and keywords.
- Fuzzy matching and metadata comparison: MLforSEO’s metadata optimisation notebook sets the stop word language to match the site.
- Keyword term counts from autocomplete suggestions.
Example
MLforSEO’s SERP analysis guide recommends running n-grams on ranking titles twice: once with stop words kept, to see patterns such as “what is”, “how to” and “vs” that signal intent and format, and once with them removed, to see topics. Patterns made entirely of stop words vanish if you only do the second run.
Limitations and things to watch
- Removing stop words can destroy meaning: “to be or not to be”, “not good”, or “flights to London” vs “flights from London”.
- Modern embedding and transformer models work better with natural sentences; MLforSEO only strips pure noise (URLs, emails, boilerplate) before BERTopic embeddings and applies stop words in the topic word step instead.
- Default lists differ between libraries and have known oddities, so check the list you use.
- Search engines largely stopped ignoring stop words in queries; do not remove them from titles or copy for “SEO” reasons.
Learn it on MLforSEO
Where Stop Words shows up on MLforSEO
Learn it in a course
Read about it on the blog
Sources and further reading
Other terms from NLP (Concepts & Pipeline)
- Content ModerationUsing NLP to flag text that may be harmful or sensitive, such as toxic, insulting or violent content, so it ca…
- Corpus (Text Corpus)A corpus is the collection of text documents you analyse or train a model on, such as your site's pages, revie…
- Emotion detection/analysisAn NLP task that labels text with specific emotions such as joy, anger, sadness, fear and disgust, going beyon…
- Entity Extraction (NER)Named entity recognition: finding names of people, organisations, places, products and other things in text an…
- Entity Sentiment AnalysisNLP that scores the sentiment expressed towards each entity in a text, so one review can be positive about one…
- Lemmatisation and StemmingTwo ways to reduce words to a base form so variants count as one: stemming chops off endings, lemmatisation re…
- Lexical or morphological analysisThe first phase of NLP: splitting text into words and analysing their structure (roots, prefixes, suffixes) so…
- N-gram (Unigram, Bigram, Trigram)A contiguous sequence of n words or characters from a text: unigrams (1), bigrams (2), trigrams (3) and so on,…
- Natural Language Processing (NLP)The field of AI that enables computers to process, analyse and generate human language, from tokenisation and…