Information Retrieval

Information retrieval (IR) is the discipline behind search: given a query, find the material (usually text documents or passages) that satisfies the information need from a large collection, and rank it. Every search engine, site search tool and AI answer engine that cites sources runs an IR system before anything is shown or generated. Understanding it is how MLforSEO explains why some content is never seen by LLMs at all.

How it works

An IR system first builds an index of its collection. Google describes this as crawling, indexing and serving. At query time it represents the query and the documents in a comparable form and scores them. Lexical methods match terms, weighted by approaches such as TF-IDF; dense methods compare embeddings with cosine similarity (semantic search). A more expensive re-ranking step then reorders the shortlist.

MLforSEO models modern AI search as a five-stage retrieval funnel, run separately for each sub-query:

• Candidate retrieval: millions of chunks narrowed to thousands by keyword and vector similarity, plus simple index presence.
• Initial filtering: thousands to hundreds via entity matching, recency and basic credibility.
• Re-ranking: hundreds to tens, where deeper relevance, information density and cross-source consistency matter.
• Claim extraction: the unit changes from passage to individual statement.
• Synthesis selection: agreement across sources decides which claims enter the answer and who is named.

The first three stages are geometry (vector proximity); the last two are language, where text is actually read.

How it’s used in SEO and marketing

• Diagnosing AI search visibility: working out whether content is cut at the early gate (topic, entities, headings) or the late gate (clarity, attribution, agreement).
• Structuring content in self-contained chunks whose headings mirror the sub-questions people ask.
• Building site search or internal tools on engines such as Elasticsearch, or RAG pipelines over your own content.
• Fuzzy matching and redirect mapping, which MLforSEO describes as small IR problems with two risks: returning unwanted matches and missing the ones you need (in other words, precision and recall).

Example

Ask an answer engine “what marketing strategies work best for SaaS companies?” and it splits the question into sub-tasks: define SaaS, identify B2B channels, retrieve performance data, check credibility and recency, then synthesise. Each sub-retrieval has its own candidate pool. A long guide covering everything may win one sub-task and lose the rest to smaller pages whose headings match each sub-question exactly.

Limitations and things to watch

• Retrieval measures relevance and does not check truth: at the early gate, a factually wrong chunk on the right topic can outscore a correct chunk on the wrong topic.
• Lexical retrieval misses synonyms; dense retrieval can miss exact identifiers such as product codes. Many systems combine both.
• The funnel is MLforSEO’s model of how these systems generally work. Exact stages, thresholds and signals differ by product and are not published.
• Nothing is retrieved unless it is crawlable, indexable and allowed for that crawler.

Learn it on MLforSEO

• How modern search actually works: the five-stage retrieval funnel
• Indexed is not selected: your AI search problem isn’t ranking
• The chunk, not the page: content an AI can actually extract
• What Fuzzy Matching Is, and Where It Fits in SEO
• AI Search Optimisation & Agentic SEO (course)