Sentence Transformers (library)

Sentence Transformers is an open-source Python library for computing and training embedding models: it turns keywords, sentences, paragraphs or pages into embeddings you can compare, cluster and search. It grew out of the Sentence-BERT (sBERT) research, was created by UKP Lab and is now maintained by Hugging Face. It is the library behind many MLforSEO Colab notebooks, from keyword-to-topic mapping to BERTopic and KeyBERT.

How it works

You load a pretrained model by name from the Hugging Face Hub, pass it a list of texts, and get back one vector per text. The documentation lists more than 25,000 pretrained Sentence Transformers models on Hugging Face. The one MLforSEO uses most is all-MiniLM-L6-v2: small, free, English-only, 384 dimensions and trained on over a billion sentence pairs, which makes it fast enough for Google Colab.

The core pattern from MLforSEO’s keyword mapping notebook:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
keyword_embeddings = model.encode(keywords)

You then compare vectors with cosine similarity (the library’s default similarity function, also available through scikit-learn). As of its 2026 documentation, the library also covers cross-encoder rerankers and sparse encoders for retrieval, alongside standard embedding models.

How it’s used in SEO and marketing

• Keyword-to-topic mapping: embed keywords and topic labels, then assign each keyword to its closest topic.
• Topic modelling: BERTopic uses all-MiniLM-L6-v2 as its default English embedding model, and you can swap in another.
• Keyword extraction and clustering: KeyBERT builds on sentence embeddings to find the phrases closest to a document.
• Meta descriptions: score candidate sentences on closeness to the page and to its main query, then pick the best one.
• Internal linking: find similar, unlinked pages and the best paragraph to place a link in.
• Hybrid fuzzy matching: PolyFuzz can use Sentence Transformers models to move from character-level to meaning-level matching.

Example

In MLforSEO’s meta description tutorial, the page is split into candidate sentences of 70 to 155 characters, each is encoded with all-MiniLM-L6-v2, and the winner is the sentence closest to both the whole page and the query it ranks for, with a small bonus for appearing early. It is still extractive, but you can see and steer why each sentence was chosen (semantic similarity in practice).

Limitations and things to watch

• Model choice matters: all-MiniLM-L6-v2 is English-only, so use a multilingual model for other markets.
• Inputs beyond the model limit are truncated (256 word pieces for all-MiniLM-L6-v2); embed chunks for long pages.
• Similarity scores are not comparable across models; re-check thresholds whenever you switch.
• The first run downloads the model, and large datasets run much faster on a GPU runtime.
• The library evolves quickly, so pin a version in notebooks you share.

Learn it on MLforSEO

• Different ways to Map Keywords to topics, with Supervised and Unsupervised ML approaches
• Topic Modelling for SEO with BERTopic
• How to Automatically Generate Meta Descriptions with Python: From BERT to LLMs
• 10 Ways to Automate Internal Linking with AI
• Introduction to Machine Learning for SEO (course)