Tools & Platforms
Gensim
In shortAn open-source Python library for topic modelling and document similarity, best known for LDA, LSI, word2vec and phrase detection on large text sets.
Gensim is an open-source Python library for topic modelling and document similarity. It is best known for its implementations of LDA, LSI (the retrieval name for LSA), word2vec and Doc2Vec, and for streaming large text collections without loading them all into memory. In MLforSEO’s tutorials it is the library behind long-form topic modelling of website content.
How it works
A typical Gensim workflow has four steps:
- Clean and tokenise your documents.
- Build a Dictionary, which gives every distinct word an ID.
- Convert each document into a bag-of-words list of (word ID, count) pairs, called the corpus.
- Train a model on the corpus, for example LdaModel, LsiModel or Nmf.
Gensim trains LDA with online variational Bayes, which processes documents in chunks, and LdaMulticore spreads training across CPU cores. It also includes a CoherenceModel for scoring topic quality (such as the c_v coherence score) and a Phrases model that detects common multi-word expressions so “machine_learning” can be treated as one token.
How it’s used in SEO and marketing
- Topic modelling a site. Cluster a crawl export by topic to see coverage, overlaps and thin areas.
- Internal linking. Use each page’s topic probabilities to suggest links between pages that share a dominant topic.
- Phrase detection. Find recurring two- and three-word phrases in keyword lists or reviews before further analysis.
- Similarity search. Compare documents in an LSI or Doc2Vec space to find near-duplicates or related content.
Example
MLforSEO’s LDA notebook, tested in October 2026 with Gensim 4.4, loads a crawl export with one row per page, cleans and tokenises the body text, builds the dictionary and corpus, then trains an LDA model with eight topics and priors learned from the data. The topics are explored with pyLDAvis and scored for coherence, and each page’s topic probabilities become the input for internal link suggestions.
Limitations and things to watch
- Its classic models rely on word counts, so pre-processing choices strongly affect results.
- Bag-of-words models struggle on short text such as titles and queries; embedding-based tools like BERTopic usually do better there.
- Older tutorials break on current versions: the pyLDAvis helper module is now pyLDAvis.gensim_models, and some function arguments have changed.
- For semantic similarity, sentence-transformer embeddings now generally outperform Gensim’s older word-vector models.
Learn it on MLforSEO
Where Gensim shows up on MLforSEO
Read about it on the blog
Sources and further reading
Other terms from Tools & Platforms
- Apps ScriptGoogle's cloud-based JavaScript platform for automating Google Sheets and other Workspace apps, often used to…
- BigQueryGoogle Cloud's serverless data warehouse for storing and querying very large datasets with SQL, including Sear…
- Claude (Anthropic)Anthropic's family of large language models and AI assistant, used for writing, analysis, coding and agentic w…
- Data StudioGoogle's free dashboard and reporting tool, renamed back to Data Studio in 2026, used to visualise SEO and ML…
- ElasticsearchAn open-source search and analytics engine for indexing large amounts of text, with fuzzy, phonetic and vector…
- FuzzyWuzzy/RapidFuzz/PolyFuzzPython libraries for fuzzy string matching. RapidFuzz is the fast default, TheFuzz is FuzzyWuzzy's successor,…
- Google CloudGoogle's cloud computing platform. For SEOs it hosts the ML APIs used most often: Natural Language, Knowledge…
- Google Cloud AutoMLGoogle's tools for training custom models without writing model code; AutoML for text was retired in Vertex AI…
- Google ColabGoogle's free, browser-based Python notebook that runs code on Google's servers, so you can use ML scripts wit…
