Foundation Model

A foundation model is a large AI model trained on broad data at scale that can then be adapted to a wide range of tasks. The term was introduced in 2021 by researchers at Stanford’s Center for Research on Foundation Models in the report “On the Opportunities and Risks of Foundation Models” (Bommasani et al.). Large language models such as the ones behind ChatGPT, Gemini and Claude are the best-known examples, and the idea also covers image, audio and multimodal models.

How it works

A foundation model is built in two stages. First comes pre-training: the model learns general patterns from a huge, varied dataset, usually through self-supervision such as predicting missing or next words, so no one has to label the data by hand. Then comes adaptation: the same model is shaped for specific jobs through fine-tuning, prompting, or connecting it to your own documents with retrieval-augmented generation.

The report used BERT, DALL-E and GPT-3 as examples. Its authors chose the word “foundation” to stress that these models are central to many systems and also incomplete on their own: they are a base to build on. They also warned about homogenisation. When many applications are built on a few models, any defect in a foundation model is inherited by everything adapted from it.

How it’s used in SEO and marketing

• Choosing tools. Most AI tools marketers use are layers on top of a handful of foundation models. Knowing which one sits underneath tells you about its behaviour, cost, knowledge cutoff and data terms.
• Adapting instead of training. You will almost never pre-train a model. You adapt one with prompt engineering, few-shot examples, fine-tuning on your own labelled data, or grounding in your own content.
• On-device features. Google describes Gemini Nano as a foundation model that runs on Android devices, and Chrome provides browser-managed foundation models through its built-in AI APIs.
• Crawler decisions. Some AI companies separate crawlers that collect training data for foundation models from those that fetch pages for live answers, which affects your robots.txt choices.

Example

A marketing team uses one foundation model in three ways. With a well-written prompt it classifies keywords by search intent. With a system prompt describing the brand voice it drafts meta descriptions for review. Fine-tuned on a few thousand reviews the team labelled by hand, it tags new reviews by product issue. The pre-training is the same in all three cases; only the adaptation changes.

Limitations and things to watch

• Inherited problems. Bias, hallucinations and gaps in a foundation model carry through to every tool built on it.
• Opaque training data. Providers rarely publish the full list of what a model was trained on.
• Loose usage. People often use “foundation model” and “LLM” interchangeably. The LLMs behind today’s chat assistants are foundation models, and the category also includes image, audio and multimodal models.

Learn it on MLforSEO

• Introduction to Machine Learning for SEO
• How to Automatically Generate Meta Descriptions with Python: From BERT to LLMs
• Comparing NLP APIs for Entity and Sentiment Analysis (vs Generative AI)
• Blog and Site Architecture for SEO and Conversions in the AI Search Era, including AI crawler choices