Data Labelling (Annotation)

Data labelling, also called annotation, is the work of attaching the correct answer to each example in a dataset. A keyword gets an intent label, a review gets a sentiment, a span of text is marked as a person or organisation. Labelled data is what makes supervised learning possible, and it is how you check whether any model, including a pre-trained API, is getting your task right.

How it works

You first define a label set and written guidelines: what counts as “commercial” intent, or where an entity starts and ends. Annotators, who may be your team, crowd workers or subject experts, then label a sample. Good projects have at least two people label an overlapping subset and measure their agreement, for example with Cohen’s kappa; low agreement usually means the guidelines are unclear. The labelled set is then split into training, validation and test data (see train/test split).

Labelling can be sped up with model-assisted pre-labels that humans correct, active learning that picks the most informative examples to label next, or rules and regex for the obvious cases.

How it’s used in SEO and marketing

• Labelling keywords by search intent or category to train an intent classifier.
• Tagging pages by page type or topic for text classification or Google Cloud AutoML.
• Hand-labelling a sample of reviews to check a sentiment API before trusting it on thousands more.
• Labelling queries as user-typed or synthetic, as in MLforSEO’s Random Forest classifier notebook.
• Marking entities in your own content to benchmark NLP APIs.

Example

MLforSEO’s framework for machine learning problems starts with one question: do you have labelled data, meaning do you already know the correct answer for at least a subset of your data? If you do, supervised methods such as classification and regression are open to you. If you don’t, you either label a sample, use a pre-trained model, or switch to unsupervised methods such as clustering.

Limitations and things to watch

• Labelling is slow and expensive, and many teams discover too late that they do not have enough labelled data for a custom model.
• Labels carry the biases and inconsistencies of the people who made them. Diverse annotators and clear guidelines reduce bias.
• Imbalanced labels (few positive examples) lead models to over-predict the majority class.
• LLMs can pre-label data cheaply, but their labels still need human review on a sample.

Learn it on MLforSEO

• The Characteristics of a Machine Learning Problem
• What Machine Learning Classification Is, and Where It Fits in SEO
• Synthetic vs User Query Classification Notebook (resource).
• Introduction to Machine Learning for SEO course