Bag of Words

Bag of Words is one of the oldest ways of representing text so a machine learning model can work with it. Each document is treated as an unordered collection of words, and the only thing recorded is how often each word appears. Grammar, word order and context are thrown away, which keeps it simple and fast.

How it works

First, the text is tokenised and cleaned (lowercasing, removing stop words, sometimes lemmatising). Then a vocabulary is built from every distinct word in the collection. Each document becomes a vector with one position per vocabulary word, holding the count of that word in the document. Stack the vectors and you have a document-term matrix, which is mostly zeros because any one document uses a small part of the vocabulary.

Common variations include recording only presence or absence (binary), weighting counts with TF-IDF, or counting n-grams instead of single words to keep a little local word order. In Python, scikit-learn’s CountVectorizer builds this matrix in a couple of lines.

How it’s used in SEO and marketing

• Classic topic modelling. LDA is a bag-of-words model, which is why pre-processing does so much of the heavy lifting in MLforSEO’s LDA tutorial.
• Topic labels in BERTopic. BERTopic clusters documents with embeddings, then uses a bag-of-words step and class-based TF-IDF to pick the words that describe each topic.
• Content comparison. Counting terms across your page and the ranking pages is a quick, transparent way to spot vocabulary gaps.
• Simple classifiers. Bag-of-words features still work well for spam filters and rule-heavy text classification with limited data.

MLforSEO’s information gain explainer also lists bag of words alongside embeddings as one of the ways a page can be represented numerically before it is compared with others.

Example

Take three page titles: “red running shoes”, “running shoes for women” and “red dress”. The vocabulary is red, running, shoes, women, dress (after removing “for”). The first title becomes [1, 1, 1, 0, 0], the second [0, 1, 1, 1, 0] and the third [1, 0, 0, 0, 1]. The two shoe titles share two words, so they look similar; the dress title shares only “red” with the first.

Limitations and things to watch

• Word order is lost: “dog bites man” and “man bites dog” get identical vectors.
• Synonyms are invisible. “Cheap” and “affordable” are unrelated columns, so meaning is only captured through exact words. Embeddings solve this.
• Vectors get very large and sparse on big collections.
• It struggles on short text. In MLforSEO’s LDA experiment, topic coherence dropped from 0.66 on full page text to 0.37 on titles alone.

Learn it on MLforSEO

• Topic Modelling for SEO with LDA
• Topic Modelling for SEO with BERTopic
• What Is Information Gain in SEO?
• Introduction to Machine Learning for SEO course