Confidence Score (Generative AI)

A confidence score is the number a model attaches to a prediction to show how sure it is. In classification it is usually a probability between 0 and 1: a text is “Finance” with 0.82 confidence, an entity is a person with 0.95. MLforSEO’s rule of thumb is simple: a low confidence score is the model telling you it is guessing.

How it works

Different tools produce different kinds of score, so always check what yours means:

• Classification APIs such as the Google Cloud Natural Language API return a confidence per category (text classification) or per safety attribute (content moderation, 0.00 to 1.00).
• Entity APIs such as Amazon Comprehend and IBM Watson NLU return a confidence per detected entity.
• The Google Knowledge Graph Search API returns a resultScore, which Google describes as an indicator of how well the entity matched the request. It is unbounded, so it is read differently from a 0 to 1 probability.
• Generative models (LLMs) do not output a calibrated confidence by default. Some APIs expose token log-probabilities; a confidence a model states in its own text is not a measurement.

How it’s used in SEO and marketing

• Filtering: keep classifications or entities above a threshold and send the rest for review.
• Quality control: if most outputs have low confidence, MLforSEO suggests fine-tuning (for example with AutoML) on a small set of correctly labelled examples.
• Entity audits: track NLP confidence and Knowledge Graph match scores for your brand over time as signs of how clearly machines recognise it.
• Moderation: flag user-generated content only when the toxicity score passes a chosen level.

Example

MLforSEO’s entity-based search lessons query the Knowledge Graph Search API: a well-established entity such as Jane Austen returns a score around 11,000, while a new brand often scores under 100. The course uses that gap as a practical heuristic (above about 1,000 suggests an established, unambiguous entity) for deciding whether your brand needs recognition work before disambiguation work.

Limitations and things to watch

• High confidence is not correctness. Models can be confidently wrong, and probabilities are often poorly calibrated.
• Scores are not comparable across tools, API versions or (for Knowledge Graph scores) different queries.
• Thresholds such as “above 0.85” are practical conventions; validate them on a labelled sample of your own data.
• For LLM outputs, use structured checks (agreement across runs, validation against source data) instead of trusting a self-reported confidence.

Learn it on MLforSEO

• What Machine Learning Classification Is, and Where It Fits in SEO
• How to do Text Classification with Google’s Natural Language API in Google Sheets
• Entity-Based Search in the LLM Era: How Google Structures and Interprets Entities
• Comparing NLP APIs for Entity and Sentiment Analysis (vs Generative AI)
• AI Search & LLMs: Entity SEO and Knowledge Graph Strategies for Brands (course)