Pre-training

Pre-training is the first and most expensive stage of building a modern model: it learns general patterns from a very large dataset, such as billions of web pages or millions of images, before anyone points it at a specific task. The result is a pre-trained model, which you can use as it is, fine-tune for your own task, or call through an API. Almost every model used in MLforSEO’s Colab practicals is pre-trained.

How it works

During pre-training, the model usually teaches itself from unlabelled data. A text model hides or predicts words: BERT was pre-trained on unlabelled text by predicting masked words using context on both sides, and GPT-style models predict the next token. An image model learns to recognise visual features across millions of photos. This adjusts billions of parameters and needs large amounts of compute, which is why it is done by big organisations such as Google, Meta, OpenAI or Alibaba.

After pre-training, models are adapted. The BERT paper showed a pre-trained model can be fine-tuned for many tasks by adding one output layer. Chat models go through post-training (instruction tuning and feedback) so they follow prompts; the Qwen3 model card lists “Pretraining & Post-training” as its two stages. The knowledge a model has without search access is fixed at the end of pre-training, which is where its knowledge cutoff comes from.

How it’s used in SEO and marketing

MLforSEO frames the choice as pre-trained, self-trained or fine-tuned, and for most marketers the answer is almost always pre-trained. These models are excellent for general tasks and common topics and fast to implement with minimal data or expertise:

• KeyBERT uses pre-trained BERT models to extract keyphrases for keyword clustering.
• The Google Cloud Natural Language API and Amazon Comprehend are pre-trained for entity extraction, sentiment and classification.
• A zero-shot transformer classifies search intent with no training data because it leans on a pre-trained model.
• Pre-trained image models (CNNs like ResNet, or CLIP) turn product images into features for clustering or labelling.

Example

In the intent classification practical, a pre-trained BART zero-shot classifier labels around 1,700 keywords in under three minutes on a GPU. It is less accurate out of the box than the entity-based approach, precisely because it relies on what it learned in pre-training, and fine-tuning on your own labelled keywords improves it.

Limitations and things to watch

• Pre-trained models are weaker in niche or fast-moving domains: specialist terms may be classified as “other” simply because they were rare in the training data.
• They inherit the gaps and biases of their pre-training data, and you usually cannot inspect that data.
• Check the licence and the model card before using a downloaded model commercially.
• If your domain is truly unique, plan for fine-tuning with a small labelled dataset before considering training from scratch.

Learn it on MLforSEO

• The Characteristics of a Machine Learning Problem explains pre-trained versus self-trained versus fine-tuned models.
• Search-Intent Classification with Machine Learning: The Approaches Explained uses a pre-trained zero-shot classifier.
• How to do keyword clustering with KeyBERT builds on pre-trained BERT models.
• Introduction to Machine Learning for SEO uses pre-trained models in every practical.