Skip to content

Tools & Platforms

Gensim

In shortAn open-source Python library for topic modelling and document similarity, best known for LDA, LSI, word2vec and phrase detection on large text sets.

Updated 24 Oct 2026· 2 min read

Gensim is an open-source Python library for topic modelling and document similarity. It is best known for its implementations of LDA, LSI (the retrieval name for LSA), word2vec and Doc2Vec, and for streaming large text collections without loading them all into memory. In MLforSEO’s tutorials it is the library behind long-form topic modelling of website content.

How it works

A typical Gensim workflow has four steps:

  1. Clean and tokenise your documents.
  2. Build a Dictionary, which gives every distinct word an ID.
  3. Convert each document into a bag-of-words list of (word ID, count) pairs, called the corpus.
  4. Train a model on the corpus, for example LdaModel, LsiModel or Nmf.

Gensim trains LDA with online variational Bayes, which processes documents in chunks, and LdaMulticore spreads training across CPU cores. It also includes a CoherenceModel for scoring topic quality (such as the c_v coherence score) and a Phrases model that detects common multi-word expressions so “machine_learning” can be treated as one token.

How it’s used in SEO and marketing

  • Topic modelling a site. Cluster a crawl export by topic to see coverage, overlaps and thin areas.
  • Internal linking. Use each page’s topic probabilities to suggest links between pages that share a dominant topic.
  • Phrase detection. Find recurring two- and three-word phrases in keyword lists or reviews before further analysis.
  • Similarity search. Compare documents in an LSI or Doc2Vec space to find near-duplicates or related content.

Example

MLforSEO’s LDA notebook, tested in October 2026 with Gensim 4.4, loads a crawl export with one row per page, cleans and tokenises the body text, builds the dictionary and corpus, then trains an LDA model with eight topics and priors learned from the data. The topics are explored with pyLDAvis and scored for coherence, and each page’s topic probabilities become the input for internal link suggestions.

Limitations and things to watch

  • Its classic models rely on word counts, so pre-processing choices strongly affect results.
  • Bag-of-words models struggle on short text such as titles and queries; embedding-based tools like BERTopic usually do better there.
  • Older tutorials break on current versions: the pyLDAvis helper module is now pyLDAvis.gensim_models, and some function arguments have changed.
  • For semantic similarity, sentence-transformer embeddings now generally outperform Gensim’s older word-vector models.

Learn it on MLforSEO

Where Gensim shows up on MLforSEO

Sources and further reading

Other terms from Tools & Platforms

See all 26 Tools & Platforms terms →