10 Ways to Automate Internal Linking with AI

Internal linking is one of the most automatable jobs in SEO, and one of the easiest to automate badly. The finding part (which pages are related, which ones nobody links to, where a link would fit) is pattern recognition at scale, which is exactly what machine learning is good at. The deciding part (does this link genuinely help the person reading this paragraph?) is not.

In this guide, I’ll walk you through ten ways to automate internal linking with AI, from orphan detection and Search Console data to LLM-generated anchor text and link health checks, and show you how to wire them into a workflow that runs on a schedule while a human keeps the final say. Every method comes with code you can run, tested on a real blog in October 2026, including a live run of the LLM anchor step with real costs and failure modes.

Looking for the strategy behind it? The audit framework (why, analyse, cluster, discover, link, deliver) comes from my BrightonSEO talk, and I’ve written it up, with the video and slides, on lazarinastoy.com. This guide assumes you know why you’re linking and focuses on how to automate it.

This guide covers:

• the automation pipeline, and the one rule that keeps it safe
• Way 1: orphan and weak-page detection
• Way 2: clustering your pages into hubs
• Way 3: embeddings-based link suggestions, down to the paragraph
• Way 4: link opportunities from Search Console query data
• Way 5: LLM anchor text generation, with guardrails (and a live test)
• Way 6: link health: broken, redirected and non-canonical links
• Way 7: embedding-powered related posts
• Way 8: Screaming Frog’s semantic similarity and AI features (no code)
• Way 9: agentic and scripted workflows
• Way 10: measuring whether any of it worked

The automation pipeline (and the one rule)

Before we get into the ten methods, here’s how they fit together. Think of internal link automation as a pipeline with one manual gate in the middle:

The rule: automate the finding, not the deciding. Similarity is not relevance to the reader. Two pages can be 0.85 similar and still make a pointless link, and an LLM can write a perfectly fluent anchor that promises something the target page doesn’t deliver. Everything below is designed so that a machine does the tedious 90%, and you spend your time on the 10% that needs judgement.

Way 1: Detect orphan and weak pages automatically

An orphan page has no internal links pointing to it. Google may still find it through your XML sitemap or external links, but your own site gives it no signal about where it fits or how important it is. Google’s own guidance is that every page you care about should have a link from at least one other page on your site (link best practices).

The trick to detecting orphans is that you can’t find them by crawling alone: a crawler only finds pages that are linked. You need a second list of URLs that should exist, and then compare. The usual sources:

• your XML sitemap (the URLs you’ve told search engines about)
• a Search Console Pages export (URLs Google shows in search, including ones you forgot about)
• GA4 landing pages (URLs that get traffic)

And it’s worth going beyond “zero links”. In practice, I flag four levels:

The no-code way: in Screaming Frog, connect Search Console, GA4 or your XML sitemaps before you crawl, run crawl analysis, then go to Reports > Orphan Pages. The Unique Inlinks and Crawl Depth columns in the Internal tab cover weak and deep pages, and Bulk Export > Links > All Inlinks gives you each link’s Link Position (content, navigation, footer), which is how you separate contextual links from the rest.

The scripted way: the automation script does the comparison for you, from either a Screaming Frog export or a WordPress site’s public REST API:

python automate_internal_linking.py orphans \
    --site https://www.example.com \
    --sitemap https://www.example.com/post-sitemap.xml

# or, with a Screaming Frog crawl and a Search Console export:
python automate_internal_linking.py orphans \
    --sf-pages custom_extraction_all.csv --sf-inlinks all_inlinks.csv \
    --sitemap https://www.example.com/sitemap_index.xml --gsc-pages gsc_pages.csv

Each URL in the report gets a status (orphan, no contextual inlinks, weak, deep, or crawled but missing from the sitemap) and the source it was found in. If you feed it a site’s REST API rather than a crawl, remember that the API only sees links in post bodies, so “orphan” there means “no in-content links”, not “no links at all”. The sitemap check is the one to schedule rather than run once: pages that are live but missing from the sitemap, or in the sitemap but linked from nowhere, tend to appear quietly after a migration or a batch of new posts.

Way 2: Cluster your pages into hubs

Hub-and-spoke (or pillar-and-cluster) linking only works if you know what your hubs are. On a 50-page blog you can do that in your head. On a 5,000-page site, you can’t, and your categories and tags usually reflect how the site was built, not what the content is about.

The companion notebook automates this in five steps:

• Embed every page. An embedding model turns each page (title plus the first few hundred words) into a vector that captures its meaning. I use the open-source BAAI/bge-small-en-v1.5 model: small, fast, good, and free to run on Colab’s CPU.
• Cluster the vectors. By default, k-means, with the number of clusters chosen automatically using the silhouette score (how well each page fits its own cluster compared with the next-nearest one). On big, messy sites, switch to HDBSCAN, which is allowed to say “this page doesn’t belong to any cluster”.
• Name each cluster with c-TF-IDF: the words that are frequent in this cluster and rare in the others, the same trick BERTopic uses.
• Pick a pillar candidate per cluster: the page closest to the cluster’s centre, nudged towards pages that already receive more internal links (70% centrality, 30% existing inlinks).
• Find bridge pages: pages almost as close to a second cluster as to their own. These are your natural places for links between hubs.

On the MLforSEO blog it found 18 clusters in 68 posts (entity SEO, NLP APIs, semantic keyword research, AI search retrieval, search intent, fuzzy matching and so on). Embedding all 68 posts took about four seconds on a laptop.

Once you have hubs, you can turn your linking strategy into rules a script can check: every spoke links to its pillar, the pillar links to every spoke, and bridge pages link across. Then the question “which links are missing?” becomes a simple comparison against your crawl.

Two cautions. Clusters are a starting point for a human, not a sitemap: give them a five-minute sanity check before you build rules on them. And very high similarity (0.95 and up) isn’t a cluster, it’s a duplicate. The notebook lists those pairs separately, because the fix is consolidation and a 301, not a link. If you want to go deeper on the topic modelling side, my BERTopic tutorial and introduction to clustering for SEO are good next steps.

Way 3: Embeddings-based link suggestions, down to the paragraph

This is the core of most AI internal linking tools, and it’s simpler than it sounds. If two pages have similar embeddings, they’re about similar things. If they’re similar and not linked, that’s a candidate.

The page-level version (which the notebook does) has a weakness: it tells you that page A should link to page B, but not where. On a 3,000-word guide, that’s the tedious part. So the script goes one level deeper and embeds every paragraph too:

• Score every page pair with cosine similarity and keep the pairs that aren’t linked yet.
• Use a relative threshold, not a fixed one. Different models score differently (bge-small rates almost every pair above 0.6; other popular models rarely go that high), so the script keeps pairs in the top 10% of your own site’s similarity distribution.
• Skip near-duplicates (0.95 and up). They need consolidating, not linking.
• Find the best paragraph: embed every paragraph of the source page and pick the one closest to the target page, skipping paragraphs that already link to it.
• Prioritise the pages that need links most: targets with the fewest inlinks get first pick, so your orphans from way 1 are served before pages that are already well linked.

python automate_internal_linking.py suggest --site https://www.example.com --brand "example" "example brand"
# -> link_suggestions.csv: source, target, the paragraph, a suggested anchor, scores, and an empty `approve` column

The heart of the passage matching is a few lines of numpy:

# emb: one normalised vector per page; chunk_emb: one per paragraph
sim = emb @ emb.T                                   # cosine similarity between pages
threshold = np.percentile(sim[np.triu_indices(len(sim), 1)], 90)

for i, j in candidate_pairs:                        # similar, unlinked, not duplicates
    ids = paragraphs_of[i]                          # paragraphs of the source page
    best = ids[np.argmax(chunk_emb[ids] @ emb[j])]  # the paragraph closest to the target

On a 79-post blog, the script embedded about 5,400 paragraphs and produced a review file of 128 suggestions that passed the anchor guardrails (more on those in way 5) in under two minutes on a laptop CPU. Two limits keep that file useful: at most three new links per source page, and at most three new links to any one target page per batch, so a single under-linked page can’t soak up every suggestion.

Academy members also have a LinkBERT notebook for internal linking opportunity discovery, which approaches the same problem with a model trained on how links appear in text.

Way 4: Find link opportunities in your Search Console data

Embeddings tell you which pages are about the same thing. Search Console tells you something embeddings can’t: which queries Google already shows each page for, and how close it is to the top. A page sitting at position 8 for a query with real impressions is a good link target, and the query itself is a ready-made anchor, written in the words people actually search with rather than a model’s guess.

The method is old (SEOs have done it with a site: search and a spreadsheet for years), but it automates very well:

• Export page + query data. The Performance report in the Search Console interface won’t give you both dimensions in one export, so use the Search Console API, the Search Analytics for Sheets add-on, or the bulk data export to BigQuery. You need page, query, clicks, impressions and position.
• Keep the striking-distance queries: positions 4 to 20, with enough impressions to matter (50 by default), and 2 to 7 words long so the query can work as an anchor.
• Check for cannibalisation first. If several of your pages rank for the same query, links should go to the one that does best, and the others need a decision (merge, differentiate or leave alone). The script picks the best-ranking page and flags the query.
• Search every other page for the query, paragraph by paragraph, and keep the ones that mention it but don’t link to the ranking page. The anchor is the source page’s own wording, so nothing is rewritten.
• Run the same guardrails as every other suggestion (more on those in way 5), and sort by impressions, so the biggest opportunities are at the top of your review file.

python automate_internal_linking.py queries \
    --site https://www.example.com \
    --gsc-queries gsc_page_query.csv --min-impressions 50 --min-position 4 --max-position 20
# -> query_link_suggestions.csv: source, target, query, impressions, position, the paragraph, the anchor

The output uses the same columns as suggest, so the same approve column and the same apply command work on it. On a sample export for a 79-post blog, it turned 242 striking-distance queries into 110 suggestions in a few seconds; no embeddings needed.

Three cautions. Search Console leaves out rare queries for privacy reasons, so this finds your head and mid-tail opportunities, not the long tail. Exact matching misses paraphrases: fuzzy matching or the passage embeddings from way 3 widen the net. And it’s the method most likely to tempt you into exact-match anchors from every page on the site, which is why the anchor-diversity guardrail caps the same anchor to the same page at two uses. If you prefer no code, Screaming Frog’s custom search (with bulk upload since version 21) can search a whole crawl for a list of queries, and the Search Console integration adds the clicks and positions next to each URL.

Way 5: LLM anchor text generation, with guardrails

Once you know the paragraph, you need the anchor: the words in that paragraph that become the link. There are two ways to automate this.

Extractive (no LLM). Take every two- to six-word span in the paragraph that doesn’t start or end with a stop word or cross punctuation, embed them, and keep the one most similar to the target page’s title. It’s free, fast and can never invent text. It’s also clumsy: when I read through the results of my test run, some anchors were exactly what I’d have picked (“semantic keyword research”, “entity recognition”), and others were awkward fragments of a sentence. A good shortlist; not a finished job.

LLM. Give a model the paragraph and a short description of the target page, and ask it to pick the best span. LLMs are much better at choosing an anchor that reads naturally and tells the reader where they’re going. They also come with risks that matter here:

• Rewriting your copy. Ask for “anchor text” and many models will helpfully rephrase the sentence to fit a keyword. Now your link insertion is also an unreviewed content edit.
• Inventing text that isn’t in the paragraph at all.
• Over-optimising: the same exact-match anchor to the same page from twenty posts. Google’s link guidance explicitly warns against cramming keywords into anchor text, and calls keyword stuffing a spam policy violation.
• Chaining links next to each other, which Google also advises against.

So the LLM never gets the last word. Every anchor, whichever method produced it, has to pass a set of guardrails, and the script logs the reason for every rejection:

Here’s the core of the verbatim check, the guardrail that matters most:

def check(anchor, paragraph):
    reasons = []
    if not re.search(r"(?<![\w])" + re.escape(anchor) + r"(?![\w])", paragraph, flags=re.IGNORECASE):
        reasons.append("not found verbatim in the passage (the model rewrote or invented text)")
    if not 2 <= len(anchor.split()) <= 7:
        reasons.append("length")
    if anchor.lower() in GENERIC_ANCHORS:
        reasons.append("generic anchor")
    return not reasons, reasons

And the instructions the model gets, which do half the guarding before the code does the rest:

Pick the span of the passage that would make the best anchor text for a link to the target page.
Rules: copy the span exactly as it appears in the passage, character for character; never rewrite,
add or invent words. 2 to 7 words; don't start or end the span with an article, preposition or
conjunction (a, the, of, for, and...). It must describe what the target page is about, so a reader knows
where the link goes. No generic phrases (click here, read more, this article) and no brand name on its own.
If no span in the passage is a natural, honest anchor for this target, return an empty anchor.

Three design choices are worth copying, whatever model you use:

• Structured output. The script asks for JSON (anchor, confidence, a one-line reason) using the API’s JSON schema output, so there’s nothing to parse by hand.
• Permission to say no. “Return an empty anchor” is in the prompt on purpose. A model forced to always answer will always find something.
• A fallback and a cost cap. If the call fails or the anchor fails the guardrails, the extractive anchor is kept and the rejection is logged. --llm-limit caps the calls per run (50 by default), and the script prints the tokens used and an estimated cost at the end of every run.

export ANTHROPIC_API_KEY=...        # pip install anthropic
python automate_internal_linking.py suggest --site https://www.example.com --llm --llm-limit 50

The script uses the Anthropic API (Claude) by default; swapping in another provider means changing one function.

What happened when I ran it for real

I ran the LLM path against the live API on a real 79-post blog, on the first 12 suggestions in the queue (the ones pointing at the least-linked pages), with two models: Claude Opus 5.5, the script’s default, at low effort, and the much cheaper Claude Haiku 4.5 for comparison. Here’s what came back:

So the whole 128-suggestion batch would cost well under a dollar on the default model. Money isn’t the constraint here; your review time is.

Where the LLM beat the extractive method. For a link to a post about what a good semantic keyword universe looks like, the extractive method offered “build a semantic model”; Claude picked “build a semantic model of your market” from the same sentence, which says more about where the link goes. Linking the other way, the extractive pick was “reshaping the keyword landscape” and Claude’s was “semantic keyword universe”: the second is the target’s actual topic. When the right anchor was obvious, the two methods agreed (“Query Context”, and a sentence that already used the target post’s title).

Where it failed, and how the guardrails caught it:

• Ignoring the length rule. Haiku returned three anchors of 11 to 22 words, a whole clause in one case, despite “2 to 7 words” in the prompt. The length guardrail rejected all three. Opus stayed within the limit but once returned eight words.
• Tidying the copy. Haiku “copied” a phrase with a straight apostrophe where the page had a curly one. The verbatim check rejected it, which is correct in spirit (the model changed the text) but wasted a good anchor. The script now maps quotes, dashes and spacing back to the page’s own characters before checking; words still have to match exactly.
• Ragged edges. Both models liked spans such as “a semantic keyword universe you can act on” or “which stage drops your content”: the stop-word rule rejected them. The script now drops a leading “a”, “an” or “the” itself (the rest is still a verbatim span), and the prompt now spells the rule out.
• Low-confidence answers. In the first run, Opus offered “Traditional keyword research” as the anchor for a post about semantic keyword research, rated its own confidence 0.4, and explained that the span named the opposite approach. Every guardrail passed it, because guardrails check form, not meaning. The script now rejects anything the model itself rates below 0.5 (--llm-min-confidence).
• Honest refusals. Opus said “no natural anchor” twice, both times correctly (the best paragraph talked about something adjacent to the target). Haiku never declined: it always found something, which is exactly what “permission to say no” is there to prevent.

Two anchors that passed everything were still ones I wouldn’t publish: “Semantic analysis” (too broad for a post about a keyword research workflow) and “keyword universe should evolve” (a fragment). That’s a 5-in-7 hit rate for anchors I’d use as they are, which is better than the extractive method but nowhere near “publish without reading”. The live run also surfaced a problem upstream of the model: because pages with the fewest inlinks get first pick, half of the first dozen suggestions pointed at the same page. The script now caps new links per target at three per batch (--max-per-target).

My takeaway: use the default model at low effort for anchors (it’s cheap enough that saving a few cents isn’t worth the extra rejections), always keep the extractive anchor as the fallback, and read every anchor in its sentence before you approve it. Every LLM answer, accepted or not, goes into a _llm_log.csv file next to the suggestions, so you can see what the model proposed and why it was rejected.

Way 6: Keep links healthy: broken, redirected and non-canonical URLs

Links decay. Posts get merged, slugs change, categories move, and every internal link that pointed at the old URL now either costs a redirect hop or ends in a 404. Internal links are the only links you fully control, so they should point straight at the live, canonical URL. Google says this directly: “When linking within your site, link to the canonical URL rather than a duplicate URL” (consolidate duplicate URLs).

Not every step in this guide needs machine learning, and this one doesn’t. But it’s what keeps the links from ways 3 to 5 pointing somewhere useful a year from now, and it’s the check you want an agent to run after every migration. The linkcheck command fetches each internal link target once and records four things: the first status code, where it ends up after redirects, how many hops it took, and the canonical URL the final page declares. Then it sorts each link into one of four problems:

python automate_internal_linking.py linkcheck --site https://www.example.com --delay 0.2
# -> link_health.csv: source, anchor, old_href, issue, fix_href, and an empty approve column
python automate_internal_linking.py apply --site https://www.example.com --suggestions link_health.csv   # dry run

The href swap leaves the anchor text alone and keeps any #section fragment. Keep your 301s in place after a migration (other sites still link to the old URLs), but stop relying on them for your own links. In Screaming Frog, the same audit lives in Response Codes > Internal (the Redirection and Client Error filters), Bulk Export > Response Codes > Internal > Redirection (3xx) Inlinks, Reports > Redirects > Redirect Chains, and the Canonicalised filter in the Canonicals tab.

One related trap for automated suggestions: near-duplicate pages. If two versions of a post exist, an embedding model can’t tell which one you want indexed, so the suggest command keeps only one of each near-duplicate pair as a link target, preferring the page the other declares as canonical, then the URL without a -2-style suffix, then the one with more inlinks. The duplicate itself is a consolidation job: merge it and 301 it.

Way 7: Embedding-powered related posts

Most “related posts” widgets pick their links by shared tags or categories, or by counting overlapping words. That’s why they so often recommend the post that happens to share a tag rather than the one a reader actually wants next. The page embeddings from ways 2 and 3 do this job better, and once you’ve computed them for linking, related posts cost you almost nothing extra.

The related command picks the most similar pages for every page, with two rules that matter in practice:

• No near-duplicates (0.95 and up): a near-copy of the page you’re reading isn’t “related”, it’s a consolidation job.
• Diversity: a candidate that’s almost identical to one already picked is skipped, so you don’t show three versions of the same topic.

python automate_internal_linking.py related --site https://www.example.com --k 4
# -> related_posts.csv (with each pick's similarity and inlinks), related_posts.json for your
#    template, and related_posts_embeddings.npy if you want live lookups

There are two ways to use the output. The simple one: render a static “Read next” block from the JSON when content changes (a build step, a nightly job, or a custom field per post). The scalable one: load the vectors into a vector database (pgvector in Postgres, or a hosted one) and look up neighbours at request time. That same index can then power on-site semantic search or a retrieval-augmented (RAG) assistant that answers questions from your own pages, so the work you do for linking pays off twice.

Keep expectations right, though. A widget link is a template link: in way 1’s terms it doesn’t count as a contextual link, and it can’t replace a link in the body copy that explains why the reader should click. Use related posts as a complement to ways 3 to 5, keep the list short (three or four), and if you want the widget to help your weakest pages, use the inlinks column to give under-linked pages a small boost.

Way 8: Screaming Frog’s semantic similarity and AI features (no code)

If you’d rather not touch Python, Screaming Frog’s SEO Spider now covers a lot of this pipeline. Here’s what was added, and when, from the release history:

To set up semantic similarity (full steps in Screaming Frog’s semantic similarity tutorial):

• Choose an AI provider under Config > API Access > AI (OpenAI, Gemini or Ollama), add the embeddings preset from the prompt library, and connect.
• Enable Store HTML and Store Rendered HTML under Config > Spider > Extraction.
• Turn on embeddings under Config > Content > Embeddings.
• Crawl, then run Crawl Analysis > Start.

What you get, and how to use it for linking:

• Semantically Similar filter (Content tab), with the closest similar URL and its score for every page. Screaming Frog itself suggests using it to improve internal linking between similar content.
• Low Relevance Content filter, for pages that don’t fit the rest of the site (off-topic pages rarely make good link targets).
• Content Cluster Diagram (Visualisations), which plots your pages by embedding. Turn on Show Inlinks Within Cluster to see whether a page gets links from the pages most similar to it. Clusters with no links inside them are your quickest wins.
• Semantic search across your crawl, for finding every page that talks about a topic, even in different words.

Two things to watch. First, the default “semantically similar” threshold is 0.95, which is tuned for finding near-duplicates. For link opportunities, lower it. Second, embeddings through a paid API cost money per page; Ollama runs locally for free but needs a decent computer.

The AI integration does more than embeddings. With a custom prompt you can, for example, ask the model to list each page’s main topics or entities as it crawls, which gives you anchor text candidates per page. And if you want the best of both worlds, you can export the embeddings Screaming Frog generated and point the notebook at them (SF_EMBEDDINGS_COL), so you get the no-code crawl and the coded clustering on the same vectors.

Version 24’s uncrawlable links filter deserves a mention too: it flags internal “links” that aren’t proper <a href> elements (span and div links, onclick handlers, JavaScript navigation). There’s no point automating link suggestions if your templates build links Google may not treat as links.

Way 9: Agentic and scripted workflows

Ways 1 to 8 are tasks. Way 9 is about running them without you having to remember to. There are three levels, and most teams should start at the first.

Level 1: a scheduled script

The simplest automation is a script that runs every week and hands you a review file. The code hub includes a GitHub Actions workflow that runs orphans, suggest and linkcheck every Monday and attaches the CSVs to the run:

on:
  schedule:
    - cron: "17 6 * * 1"        # Mondays 06:17 UTC
jobs:
  suggest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python automate_internal_linking.py orphans --site https://www.example.com --sitemap https://www.example.com/post-sitemap.xml
      - run: python automate_internal_linking.py suggest --site https://www.example.com
      - run: python automate_internal_linking.py linkcheck --site https://www.example.com
      - uses: actions/upload-artifact@v4
        with: { name: internal-link-review, path: "*.csv" }

If you live in Screaming Frog, the equivalent is a scheduled crawl with Auto Compare Crawls (version 24), which compares the last two crawls and emails you what changed, including new orphans.

Level 2: review, then apply

The apply command closes the loop for WordPress. It reads only the rows you marked yes, fetches each post’s raw content with an application password, links the first plain-text occurrence of the anchor inside a paragraph or list item (never inside an existing link, a heading, code or a button, and never if the post already links to that target), and prints a diff. Rows from linkcheck work the same way: it swaps the old href for the fixed one and leaves the anchor alone. Nothing is saved until you add --write, and WordPress keeps a revision of every change, so you can roll back.

python automate_internal_linking.py apply --site https://www.example.com              # dry run: prints the diff
python automate_internal_linking.py apply --site https://www.example.com --write      # saves

Test it on a staging site first. The insertion logic is deliberately conservative, and I tested it against sample block markup, not against your theme’s quirks.

Level 3: an agent

With Screaming Frog’s MCP server (version 24), an AI assistant such as Claude can run crawls, analyse them and export data from a chat window. Combine that with the script as a tool, and an agent can do the whole loop: crawl, detect orphans, suggest links, draft anchors and prepare the review file, then come back to you with a summary.

Here’s the brief I’d give an agent, and the boundaries I’d hold it to:

Crawl https://www.example.com/blog/ with the SEO Spider, export All Inlinks and the content extraction,
then run automate_internal_linking.py orphans, linkcheck and suggest (with --llm, --llm-limit 50).
Summarise: new orphans since last week, new broken or redirected links, the 20 highest-priority
suggestions, and any duplicate pages. Do NOT run apply --write. Stop and ask me before changing anything on the site.

The boundaries matter more than the brief. An agent should never publish, delete or redirect on its own, and the step that writes to your CMS should always sit behind a human approval. Screaming Frog is upfront that not every feature is supported in its MCP yet, so check what your agent can actually see before you trust its summary.

Way 10: Measure whether it worked

Automation makes it very easy to add 150 links in an afternoon. It doesn’t tell you whether they helped. Internal link effects are notoriously hard to predict, and SearchPilot, who run SEO split tests for a living, say it plainly: it’s “always wise to test site changes even when they seem like obvious wins” (lessons from their internal link tests).

A measurement set-up that a script can run for you:

• Benchmark before you change anything: inlinks, contextual inlinks, orphans and crawl depth from way 1, plus clicks, impressions and positions per page (the same Search Console data as way 4).
• Split comparable pages into a test group (gets the new links) and a control group (doesn’t). Same template, similar traffic, same section.
• Wait for recrawling. Skip the first week or so after the change.
• Compare the change in both groups (difference in differences), so seasonality and algorithm updates, which hit both groups, cancel out.
• Measure both sides of the link: the pages that received links and the pages that gave them. SearchPilot measures both, because adding links can change the linking page too.

python automate_internal_linking.py measure \
    --gsc-daily gsc_pages_by_date.csv \
    --changes changed_pages.csv        # url, change_date

# clicks: test +12.1% vs control +0.3% -> difference in differences +11.8 points

That output line is from a synthetic dataset where I planted a 15% uplift for the test pages partway through the window, to check the script finds it (it does, diluted by the days before the uplift kicked in). On real data, expect noisier numbers, and treat the result as a sanity check, not proof: small groups and other releases move it too. For a proper answer, randomise test and control before you add links, or use a split-testing platform.

For ongoing reporting, group pages and queries by topic cluster in Data Studio (custom dimensions do this nicely; I’ve written up the formulas on Data Studio for Marketers), and use GA4’s path exploration to check whether people actually move from informational pages to the high-intent pages you linked.

The ten ways at a glance

Get the code

Everything in this guide is in the internal-link-audit-topic-clustering folder of the MLforSEO code hub:

• automate_internal_linking.py: the orphans, suggest, queries, linkcheck, related, apply and measure commands, with the guardrails
• the audit and clustering notebook: link metrics, anchor classification, hubs, pillar candidates, bridge pages (Colab, no API keys)
• examples/weekly-internal-link-suggestions.yml: the scheduled GitHub Actions workflow
• example outputs, including the log of the live LLM anchor test, so you can see what you’ll get before you run anything

And if you want the framework that sits behind all of this, from the aims of an internal linking strategy to what goes in the deliverable, read my BrightonSEO talk write-up.

Related glossary terms

• Embedding
• K-Means Clustering
• HDBSCAN
• c-TF-IDF
• LinkBERT
• N-gram
• Entity Extraction (NER)
• Fuzzy Matching
• Similarity Score