Keyword Clustering: SERP-Based vs Semantic, and a Free Tool for Both

Keyword clustering tools usually make you pick a side, often without telling you. Some group keywords by meaning: they turn each keyword into numbers and put similar ones together. Others group keywords by what Google shows: two keywords belong together if their results pages share enough URLs. Both are called “keyword clustering”, both produce tidy-looking groups, and they regularly disagree.

I teach both, and my honest view is that neither is right on its own. Meaning-based clusters are cheap and fast but can’t see intent. SERP-based clusters see intent but cost money and leave gaps. So I built a free notebook that runs both, shows you exactly where they disagree, and combines them into a hybrid.

This post compares the two approaches, uses the notebook’s disagreement tables as worked examples, places KeyBERT and BERTopic alongside them, and ends with my recommendation for when to use which. One important note up front: the notebook’s sample data is a synthetic home-coffee keyword set, with invented volumes and made-up SERPs on fictional .example domains. Every number below comes from that sample run. It shows how the methods behave; it isn’t evidence of how any real SERP looks.

What keyword clustering is for

Keyword clustering groups a keyword list into sets that one page could target. It’s the step between a messy export of a few thousand keywords and a keyword map: one cluster, one page, one primary keyword, and the rest as secondary keywords and subheading ideas.

Under the hood it’s unsupervised learning: there are no labels, so the algorithm decides what “similar” means based on whatever signal you give it. That choice of signal is the whole debate. If you want the broader picture first, read what machine learning clustering is and where it fits in SEO.

Semantic clustering: grouping by meaning

Semantic clustering turns every keyword into an embedding, a list of numbers that captures its meaning, and groups keywords whose embeddings are close. The notebook uses all-MiniLM-L6-v2, a small, free Sentence-BERT model that runs fine on a CPU (Sentence-BERT was introduced in Reimers and Gurevych, 2019). Closeness is measured with cosine similarity.

The grouping is hierarchical (agglomerative) clustering: it keeps merging the closest groups until the next merge would join groups further apart than a distance threshold (0.40 by default). I like this because you don’t have to guess the number of clusters up front, which is the weak spot of k-means for keyword work (see the scikit-learn documentation for how the linkage works).

On the 160 sample keywords, this produced 62 clusters. The notebook also runs HDBSCAN as an alternative, which found 2 clusters and left 15 keywords as noise. On short keyword lists HDBSCAN tends to either lump most keywords into a few huge groups or leave many unassigned, which is why it isn’t the default.

What semantic clustering sees: wording and meaning, across any list size, for free. What it misses: whether Google treats two similar-sounding searches as the same query. “cold brew recipe” and “best cold brew maker” are close in meaning, and the embedding step put them in the same 16-keyword cold brew cluster, yet one wants instructions and the other wants products.

Where KeyBERT and BERTopic fit

People often ask me how this relates to KeyBERT and BERTopic, which I’ve written about before. KeyBERT extracts keyphrases, and in my KeyBERT keyword clustering guide I use those keyphrases to group and label keywords. It’s great for naming themes. BERTopic is topic modelling: it shines on longer text, such as page content or reviews, and is what I’d reach for to cluster a whole site (see topic modelling for SEO with BERTopic). Both are semantic approaches. Neither looks at the SERP.

SERP-overlap clustering: grouping by what Google shows

SERP-overlap clustering asks a different question: does Google answer these two searches with the same pages? If two keywords share several top-10 URLs, that’s the clearest evidence you can get that one page can rank for both.

The notebook uses a simple, explainable hard clustering method. It takes the highest-volume keyword that isn’t in a cluster yet, adds every unclustered keyword that shares at least three top-10 URLs with it, and repeats. Every member is compared with the cluster’s head keyword, not with each other, so clusters can’t drift through long chains of loosely related keywords. URLs are normalised first (protocol, www., tracking parameters and trailing slashes removed), so the same page always counts once.

On the sample: 35 SERP clusters; 3 keywords have no SERP data (left out of this method). Those three keywords are missing on purpose, to show the fallback later.

What SERP overlap sees: intent as Google interprets it, including pairs that look nothing alike. What it misses: any keyword you don’t have SERP data for, and it reflects one snapshot for one location and device. It also costs money: you need the top 10 for every keyword.

If you don’t have a SERP export, the DataForSEO SERP Collection Starter Notebook produces exactly the keyword, rank and URL table this method needs, at about $0.002 per keyword in Live mode or $0.0006 in Standard mode (October 2026 list prices). I walk through it in how to pull SERP data with the DataForSEO API. The clustering notebook also includes a small optional fetcher for missing SERPs.

Where the two methods disagree

The coffee sample is synthetic, so I also ran the notebook on 29 real keywords around keyword clustering, semantic SEO and AI Overview tracking (live Google US desktop SERPs pulled with DataForSEO on 25 October 2026). Real SERPs in this niche overlap far less than the sample suggests: the agreement between the two methods dropped to an ARI of 0.04, against 0.37 on the sample, and the “different wording, same SERPs” table came back empty. The clearest example is “ai overview tracker” and “track ai overviews”: the embeddings rate them 0.89 alike, yet they share no URLs, because Google shows tool pages for one and how-to articles for the other. One more thing worth knowing: “top 10” here means the organic URLs on page one. AI Overviews, People Also Ask and video blocks take some of those ten slots, so you typically get seven to nine organic URLs per keyword.

The notebook scores agreement with the adjusted Rand index (ARI), where 1 means identical clusterings and 0 means no better than chance. On the sample, embeddings and SERP overlap agreed at 0.37. Related, but far from the same.

The ARI is a summary. The two disagreement tables are where the decisions are.

Similar meaning, different SERPs

These are pairs the embeddings would put together that share almost no URLs. Usually that’s a difference in intent, so they likely need separate pages.

The first row is my favourite example of why meaning alone misleads. The two phrases use almost the same words and score 0.97, but one is a search for milk and the other is a search for a gadget. The espresso rows show the same thing with intent: “best” (reviews and comparisons) against “cheap”, “deals” and “buy” (shopping). Remember that these SERPs are synthetic; the generator deliberately gave those keywords different result sets. With real data, this table is your list of pairs to check in the live SERP before you merge pages.

Different wording, same SERPs

These are pairs that look unrelated but share many URLs. One page can probably target both.

“microfoam vs macrofoam” and “how to froth milk without a frother” score 0.04 on meaning, which an embeddings-only tool would never group. If the real SERPs share eight of ten URLs, one milk-texturing guide could serve both, and splitting them would mean two thin pages competing with each other.

The hybrid: SERP evidence where you have it, meaning where you don’t

The hybrid blends both signals into one similarity score for every pair of keywords:

serp_sim = shared / SERP_TOP_N                       # shared URLs out of 10
both_have_serp = np.outer(has_serp, has_serp)
hybrid_sim = np.where(both_have_serp,
                      HYBRID_SERP_WEIGHT * serp_sim + (1 - HYBRID_SERP_WEIGHT) * emb_sim,
                      emb_sim)

When both keywords have SERP data, Google’s evidence counts for 60% and meaning for 40% by default. When either keyword has no SERP data, the score falls back to meaning alone, so nothing is dropped. The pairs are then clustered with the same agglomerative method, stopping at a distance threshold of 0.60.

On the sample this gave 39 hybrid clusters, with an ARI of 0.79 against the SERP clusters and 0.46 against the embedding clusters. In other words, with the default weight, the hybrid mostly follows Google where it has the evidence and fills the gaps with meaning. Here are the five biggest clusters by volume (synthetic volumes):

Compare the cold brew cluster with the embedding version. Meaning alone produced one 16-keyword cold brew group mixing recipes, makers and caffeine questions. The hybrid kept the seven recipe and ratio keywords together (“cold brew recipe”, “cold brew ratio”, “how long to steep cold brew” and so on) and moved the product searches elsewhere. That’s the split I’d want for a real site: a recipe guide and a buying guide.

Walking through the notebook

1. Settings and data. Run it once with USE_SAMPLE = True. Then upload your keyword CSV (keyword and ideally volume) and, if you have it, a long SERP export with keyword, rank and url. Duplicates are merged and their volumes summed.
2. Method A, embeddings. Encodes every keyword and clusters with a distance threshold. It runs on a CPU; the README reports the sample run taking under a minute, including the model download.
3. Method B, SERP overlap. Normalises URLs, counts shared URLs for every pair, and builds the head-keyword clusters. Set FETCH_SERPS = True to fill missing SERPs from DataForSEO.
4. Disagreements. Prints the ARI and the two tables above.
5. Method C, the hybrid. Blends the two similarity matrices and clusters again.
6. Naming. Adds a label, a central keyword, an intent hint and volume totals to every cluster.
7. Export. Writes keyword_clusters.csv, cluster_summary.csv and keyword_clusters.xlsx (two tabs), which you can import into Google Sheets.

How to read the results

• Label vs central keyword. The label is the cluster’s highest-volume keyword, which is usually your primary keyword. The central keyword is the one closest in meaning to all the others. When they differ, as with “best espresso machine” and “top rated espresso machines”, the central keyword is often a better angle for the H1 or the page brief.
• Mixed intent. The intent hint is a volume-weighted vote from a few simple search intent rules (informational, comparison, commercial, transactional). If intent_mix is split, say 55% comparison and 45% informational as in the “burr vs blade grinder” cluster, look at the members. You may want two pages.
• Keywords without SERP data. They’re placed by meaning alone, so they can land in the wrong cluster. In the sample, “nespresso aeroccino review” (a milk frother) has no SERP data and joined the espresso machine cluster because the words look alike. The keywords_without_serp column in the summary tells you where to look.
• Tune, then re-run. Too many tiny clusters: raise the distance thresholds. Clusters too broad: lower them, or raise HYBRID_SERP_WEIGHT for stricter, intent-led groups.

From there, the output feeds straight into keyword mapping. If you’re building the map from scratch, how to structure and categorise keyword data and how to build a search intent-first website architecture pick up where this notebook stops.

Which method should you use? My recommendation

The workflow I’d suggest uses all three in sequence: embeddings to scope the topic, SERP pulls for the keywords that matter commercially, then the hybrid to produce the map.

Pitfalls and limits

• The sample is synthetic. Invented volumes and fictional URLs. Treat every number in this post as a demonstration of the method.
• SERPs move. Overlap from one snapshot for one location and device is a strong signal, not a fact. Re-pull before big decisions.
• English by default. all-MiniLM-L6-v2 is an English model. For other languages, swap in a multilingual one such as paraphrase-multilingual-MiniLM-L12-v2.
• Thresholds are tuned on short English keywords. Look at a few clusters before you trust the defaults on your data.
• The intent hint is a handful of regular expressions. It speeds up a human review; it doesn’t replace one.

What to do next

Start with the sample, then run it on one topic area of your own with embeddings only. If the clusters look useful, pull SERPs for that topic and run the hybrid. The same SERP pulls also tell you which of those keywords show an AI Overview and whose pages it cites, which you can check with the AI Overview Tracker (walkthrough: how to track AI Overviews for your keywords).

More clustering resources: the KeyBERT for SEOs Colab template, the sBERT and BERTopic keyword-to-topic notebook, and the complete guide to ML clustering for marketing and SEO. To learn how clustering works and which model to pick when, take my Introduction to Machine Learning for SEO course.

FAQ

Is there a free keyword clustering tool?

Yes. This notebook is free and runs in Google Colab. Embeddings-only clustering needs nothing but your keyword list. SERP-overlap and hybrid clustering need SERP data, which you may already have from a rank tracker export, or can pull through DataForSEO for a fraction of a cent per keyword.

What’s the difference between SERP-based and semantic keyword clustering?

Semantic clustering groups keywords whose meaning is similar, using embeddings. SERP-based clustering groups keywords whose search results share URLs. Semantic is cheaper and covers every keyword; SERP-based reflects how Google interprets intent. On the notebook’s sample they agreed at an ARI of 0.37.

How many shared URLs should two keywords have to be clustered together?

The notebook’s default is three of the top 10. Raise it for stricter, more intent-specific clusters, and lower it for broader ones. Check a few clusters by hand whichever value you choose.

Does it work for languages other than English?

SERP overlap works in any language. For the embedding and hybrid steps, swap the model for a multilingual one, and edit the intent rules, which are English regular expressions.