Model Evaluation
Coherence Score
In shortA metric for how interpretable a topic model's topics are, based on how often each topic's top words appear together in real text.
A coherence score measures how much a topic produced by a topic model makes sense to a human. It looks at the top words that describe each topic and checks how strongly they belong together in real text. Higher coherence usually means cleaner, more interpretable topics, which makes it the standard way to compare topic models and settings.
How it works
Coherence scores take the top N words of each topic (often 10) and measure how often pairs or groups of those words co-occur, either in your own documents or in a reference corpus. The scores for all word pairs are combined into a topic score, and the topic scores are averaged for the whole model.
There are several variants. The most common in Python’s Gensim library are:
- c_v: uses a sliding window and word-vector similarity; scores usually fall between 0 and 1 and it correlated best with human judgement in Röder and colleagues’ 2015 comparison.
- u_mass: based on document co-occurrence within your own corpus; scores are negative, with values closer to zero being better.
- c_npmi and c_uci: based on pointwise mutual information between word pairs.
Because the variants use different scales, only compare scores calculated with the same measure on the same reference text.
How it’s used in SEO and marketing
- Choosing the number of topics for LDA or NMF by fitting several models and picking where coherence peaks.
- Tuning hyperparameters such as alpha and beta, often alongside perplexity.
- Comparing models, for example LDA against BERTopic, before clustering a site’s content or a keyword universe.
Example
In MLforSEO’s BERTopic tutorial, the same 1,373 short titles were modelled three ways and scored with c_v. LDA reached 0.33, while NMF and BERTopic both reached 0.41; on the full text of the posts, LDA scored around 0.66. The lesson: coherence drops sharply on very short documents, so enrich titles with headings or descriptions where you can.
Limitations and things to watch
- A high score does not guarantee useful topics for your business question; read the topics yourself.
- Scores depend on preprocessing, the number of top words and the reference corpus.
- Coherence says nothing about how well topics cover your documents or how many are left unassigned.
Learn it on MLforSEO
Where Coherence Score shows up on MLforSEO
Learn it in a course
Sources and further reading
Other terms from Model Evaluation
- Accuracy (ML metric)Accuracy is the share of all predictions a model gets right. It is the simplest evaluation metric, but it misl…
- Confusion MatrixA confusion matrix is a table comparing a classifier's predictions with the true labels, showing exactly which…
- Cosine SimilarityA measure of how closely two vectors point in the same direction, used to score how similar two texts, keyword…
- Emotion ScoresPer-emotion scores (joy, sadness, fear, disgust, anger) an emotion analysis API assigns to a text, each from 0…
- Entity Salience ScoreA 0 to 1 score from Google's Natural Language API showing how central each extracted entity is to the analysed…
- Euclidean DistanceEuclidean distance is the straight-line distance between two points in any number of dimensions, used to compa…
- F1 ScoreA single evaluation metric that balances precision and recall (their harmonic mean); useful when classes are i…
- Ground TruthGround truth is the set of correct answers, usually labelled by people, that a model is trained on or checked…
- LLM-as-a-JudgeUsing a large language model to score or compare outputs against a rubric, as a scalable stand-in for human re…