LLM-as-a-Judge

LLM-as-a-judge is an evaluation method where a large language model grades the output of another model, or of a person, against criteria you define. Instead of a human reading every generated title, summary or answer, the judge model scores each one or picks the better of two, and humans review a sample.

How it works

You write a judging prompt with a clear rubric: what to check, how to score it, and what evidence to cite. Common formats are single-answer grading (score 1 to 5 on accuracy, relevance and tone), pairwise comparison (which of A or B is better?) and reference-guided grading, where the judge compares an output with a known correct answer or the source text. Asking the judge to explain before scoring, and returning structured output, makes results easier to audit.

The method was popularised by research such as MT-Bench (Zheng et al., 2023), which reported that strong judges such as GPT-4 reached over 80% agreement with human preferences, about the same level at which humans agree with each other. Frameworks such as G-Eval use step-by-step evaluation prompts in a similar way.

How it’s used in SEO and marketing

• Quality-checking AI-generated meta descriptions, titles or alt text for length, accuracy against the page, and banned claims.
• Checking groundedness: does every fact in a summary appear in the source?
• Scoring content briefs or drafts against an editorial checklist.
• Labelling relevance of internal link or related-post suggestions at scale.
• Comparing prompts or models before switching a pipeline.

Example

A pipeline generates 500 meta descriptions. A judge prompt receives each page’s text and its description and answers three questions: is every number and claim present on the page, is the main query included, and is it under 155 characters? It returns pass or fail with the offending phrase. An editor reviews all failures and a random 5% of passes to check the judge itself.

Limitations and things to watch

• Judges have documented biases: position bias (favouring the first or second answer), verbosity bias (favouring longer answers) and self-enhancement bias (favouring outputs from their own model family).
• A judge can hallucinate or miss errors it would also make itself.
• Validate the judge on a human-labelled sample before trusting it, and re-check when you change models.
• Swap answer order in pairwise tests and average the results.
• Deterministic checks (length, regex, exact match) are cheaper and more reliable where they work; use the judge for what rules cannot assess.

Learn it on MLforSEO

• Comparing NLP APIs for Entity and Sentiment Analysis (vs Generative AI) on evaluating outputs against labelled data.
• How to Automatically Generate Meta Descriptions with Python: From BERT to LLMs on guardrails for generated copy.
• 10 Ways to Automate Internal Linking with AI on guardrails for LLM anchor text.