Skip to content
Browse courses

ML Models & Algorithms

DistilBERT

In shortA smaller, faster distilled version of BERT; in SEO it powers Mark Williams-Cook's classifier that predicts Google's eight query semantic classes.

Updated 24 Oct 2026· 2 min read

DistilBERT is a compact version of BERT released by Hugging Face in 2019. It keeps most of BERT’s language understanding while being much smaller and faster, which makes it a popular base for text classifiers. In SEO it is best known as the model behind Mark Williams-Cook’s Refined Query Semantic Class Classifier, which predicts which of eight Google query classes a keyword belongs to.

How it works

DistilBERT is built with knowledge distillation: a small “student” model is trained to reproduce the behaviour of a large “teacher” model, here BERT. According to its paper, the result is 40% smaller and 60% faster while retaining 97% of BERT’s language understanding performance. Like BERT, it is then fine-tuned on labelled examples to do one job, such as sorting queries into categories (multi-class classification).

The query classifier came out of an exploit Williams-Cook’s team found in Google’s systems, which exposed how Google labelled queries. Almost all queries were assigned one of eight “refined query semantic classes”: SHORT_FACT, BOOL, INSTRUCTION, DEFINITION, REASON, COMPARISON, CONSEQUENCE and OTHER. He fine-tuned DistilBERT on 4.8 million queries from that data and published it as a free tool.

How it’s used in SEO and marketing

  • Labelling a keyword universe by the kind of answer each query expects, then matching content format to it: definitions for DEFINITION, steps for INSTRUCTION, tables for COMPARISON.
  • Flagging SHORT_FACT and BOOL queries, which are the most likely to be answered on the results page or in an AI Overview.
  • Adding an answer-format layer on top of macro search intent labels (informational, commercial and so on).
  • Fine-tuning your own DistilBERT intent model on a small labelled sample when off-the-shelf models miss your niche’s language.

Example

Run a keyword list about Google Trends through the classifier and you would expect “is pytrends free” to come back as BOOL, “how to use pytrends” as INSTRUCTION and “pytrends vs serpapi” as COMPARISON. Those three queries need a one-line answer, a step-by-step tutorial and a comparison table respectively, even though a traditional tool would call them all informational.

Limitations and things to watch

  • The labels are the model’s prediction of Google’s class, learned from data captured at one point in time. As MLforSEO’s intent lesson notes, whether Google still uses this scheme is up for debate.
  • The eight classes describe answer format, not commercial value, so combine them with intent and business data.
  • It is trained on English queries.
  • DistilBERT trades a little accuracy for speed; for long or nuanced text a larger model may do better.

Learn it on MLforSEO

Where DistilBERT shows up on MLforSEO

Learn it in a course

Sources and further reading

Other terms from ML Models & Algorithms

See all 55 ML Models & Algorithms terms →