SERP Competitor Analysis with ML: Categories, Entities, Sentiment and N-grams (Notebook)

Before I write anything competitive, I want to know who ranks, what kind of sites they are, which topics and phrases keep showing up, and how the results feel. This free Colab notebook answers those questions for a set of keywords in one run. It is the refreshed version of the four methods I first presented at BrightonSEO in 2022.

Who it’s for

Content and SEO teams doing competitor or SERP research before a brief, and anyone who wants evidence for “what does Google reward here?” instead of a gut feeling.

How to use it

1. Get the SERP data. Use the bundled sample (a real pull of the US desktop top 10 for six machine learning keywords), upload your own CSV, or pull live results with the DataForSEO API. Snippets are cleaned of dates and “Read more”.
2. Categorise competitors. A rule-based starter taxonomy labels each domain’s site type and each result’s intent with no key. Optionally, an LLM applies your own category list to the domains the rules can’t place, using structured outputs so it can only answer with your categories.
3. Calculate share of search per domain, site type and intent, both as a share of slots and click-weighted with a CTR curve you can edit.
4. Find prominent entities with salience, sentiment and average position using the Google Cloud Natural Language API v1 (it still returns salience; v2 doesn’t). Without a key, spaCy extracts the entities.
5. Score sentiment for every title and description (Natural Language API v2) and chart it against position.
6. Analyse n-grams twice: title patterns with stop words kept (“what is”, “vs”) and topics with them removed, split by intent and position.
7. Export one tidy CSV per analysis, ready for a dashboard.

Prefer spreadsheets? ngrams.gs in the same folder adds an =NGRAMS(B2:B100, 2) custom function to Google Sheets, no Python needed.

What it costs

• Nothing on the sample data or with your own CSV: the rule-based categories, spaCy entities and n-grams need no keys.
• Google Natural Language API: the first 5,000 units a month are free per feature (a unit is 1,000 characters).
• DataForSEO live organic results cost roughly $0.002 per keyword (October 2026), and the optional LLM step is billed per token by Anthropic. Check current pricing before a big run.

Limits

• The starter taxonomy is a lookup of known domains plus URL patterns, so domains it doesn’t know come back as “unclassified” until you add them or use the LLM step.
• The Natural Language API and LLM paths were tested against documented response formats; run them once on a few keywords with your own keys before you rely on them.

Want to schedule this and add AI Overview citations, People Also Ask clustering and volatility alerts? The companion automation notebook in the same folder covers all ten methods from 10 Ways to Automate SERP Analysis with AI. SERP research is a core part of the Semantic ML-enabled Keyword Research course.