Screaming Frog SEO Spider

Screaming Frog SEO Spider is a desktop crawler, made by UK agency Screaming Frog, that crawls a website the way a search engine bot would and reports on what it finds: status codes, redirects, titles, meta descriptions, canonicals, structured data, internal links and much more. For MLforSEO it is also the quickest way to build a dataset: crawl a site, extract the text you need and feed it into a machine learning model.

How it works

You enter a URL (or upload a list), the Spider follows links and records every page’s data, and you filter or export the results. Key features for ML work:

• Custom extraction: pull any part of a page (main content, author, dates, prices) into its own column using CSS path, XPath or regex. Since version 19 you can use visual custom extraction: click the element on the page and the tool suggests the selector.
• Rendering JavaScript, so content added by scripts is captured.
• Structured data extraction and validation.
• AI and embeddings: since version 21 the Spider can send crawled content to OpenAI, Gemini or a local Ollama model, and since version 22 it can create embeddings and flag semantically similar and low-relevance pages.

The free version crawls up to 500 URLs; custom extraction and the AI features need a paid licence, which was £199 a year at the time of MLforSEO’s October 2026 update.

How it’s used in SEO and marketing

• Web scraping page content into a CSV for entity extraction, topic modelling or classification.
• Internal linking audits: orphan detection, anchor text, crawl depth and semantic similarity between pages.
• Auditing images, metadata and schema at scale before running ML on the outputs.
• Generating meta descriptions or alt text with an AI prompt during the crawl.
• Content audits that combine crawl data with Search Console and analytics through the tool’s API integrations.

Example

To topic-model a blog, set up a custom extraction on the article body’s CSS selector, crawl the blog folder, then export URL and extracted text. The companion notebook in the MLforSEO code hub tests the selector and cleans the export into an NLP-ready dataset, which can go straight into BERTopic or the Natural Language API.

Limitations and things to watch

• It only extracts from pages it can crawl that return a 200 status code; blocked or login-only pages are out of reach.
• Crawling large sites needs memory or database storage mode, and polite crawl speeds on sites you don’t own.
• AI features call external APIs, so they add cost and send your content to that provider.
• Version numbers and pricing change; check the official user guide for current menus.

Learn it on MLforSEO

• Web Scraping of Page Content in 3 Easy Steps with Screaming Frog
• 10 Ways to Automate Internal Linking with AI
• How to Automatically Generate Meta Descriptions with Python: From BERT to LLMs
• Image SEO in 2026
• Introduction to Machine Learning for SEO