Structured Data Automation with Python: Detect, Generate and Validate JSON-LD at Scale

Stop typing schema by hand. Structured data is a standardised format for providing information about a page and classifying the page content. It helps search engines understand what a page is about, and it is what makes a page eligible for rich results: the stars, prices, dates, breadcrumbs and images you see in graphical search results.

There are several benefits, both for you and for search engines, that come from implementing structured data, including:

• Better understanding: makes your site easier for search engines (and the AI systems built on them) to understand: which entities a page is about, who wrote it, what it sells. Google is clear that structured data isn’t a ranking factor in itself, but understanding is where everything else starts.
• Rich results: allows your website to qualify for richer, more informative search results.
• Increased click-through rates (CTR): rich results can grab user attention and lead to more clicks.
• Better user experience: provides context to search engines, potentially leading to more relevant results and a better user experience.

JSON-LD is Google’s preferred format for structured data. You can generate structured data with any web generator (like Rank Ranger’s or Merkle’s generator), but this requires manual data entry for each page, which becomes tedious very quickly. Today, I want to show you how you can extract the data from your website programmatically and automate the process of structured data opportunity detection, generation, implementation, and validation, with Python.

All of the code is in the MLforSEO code hub: grab the structured data automation toolkit here. It’s one Python file with three commands (audit, generate and validate) plus sample data, and it runs locally or in Google Colab.

Which structured data still earns a rich result in Google (2026)?

Before you automate anything, it’s worth knowing what you’re automating for. Automating markup that Google no longer displays is still useful for other consumers, but it shouldn’t be your first priority. Here’s where things stand, based on Google’s search gallery and documentation changelog as of 22 October 2026:

Two things haven’t changed. Retired markup does no harm, and Google has said so each time it retired a feature: it doesn’t affect ranking. And the markup has to match what’s visible on the page. Google’s guidance on AI features repeats this, and adds that you don’t need any special markup to appear in AI Overviews or AI Mode. Structured data helps machines understand your entities. It isn’t a magic AI switch.

How to automate structured data opportunity detection and generation: step-by-step project

By using a programmatic approach for the automated detection of keywords and patterns on a page, you can build a rule-based classification (labelling) system that automatically identifies signals of structured data opportunities, such as:

• headings and sections structured as question-answer pairs, and frequently asked questions sections, both of which used to signal an opportunity for FAQ structured data. FAQ rich results are gone, so today these are a flag to review rather than a quick win (more on that below);
• specific terms mentioned, like course, property, product or job, combined with a custom page-type classifier, which could signal an opportunity for adding specific structured data types like Course list, Product, Job posting, Event and many others;
• specific content types detected from the structure of the content, like news articles, video pages or how-to guides, triggering a flag to highlight an opportunity for news article schema, video schema, or (with the same caveat as FAQ) how-to schema, and so on.

The great pioneer Hamlet Batista also developed a machine learning approach that uses AutoML to detect objects on a page. This approach is more suitable for custom types of structured data, and can be very useful for experts who also want help building the databases from which they create the structured data code (as opposed to simply identifying the opportunity for it). It uses AutoML and Pyppeteer to detect and scrape the elements used to generate product structured data. If you want to rebuild it today, AutoML Vision now lives in Vertex AI, and Playwright has replaced Pyppeteer (which is no longer maintained) as the go-to headless browser for Python.

For most sites, though, you don’t need computer vision. Your pages are generated from templates and a database, so the data is already structured; it just isn’t marked up. Here’s the three-step project.

Step 1: Crawl your pages and extract the structured data that’s already there

Start with an inventory. For every URL, you want to know which structured data is already on the page, in any format: JSON-LD, Microdata or RDFa. The extruct library does exactly this:

!pip install -q extruct requests beautifulsoup4 lxml

import requests, extruct

url = "https://www.example.com/blog/some-post/"
html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).text

data = extruct.extract(html, base_url=url, uniform=True,
                       syntaxes=["json-ld", "microdata", "rdfa"])
data["json-ld"]      # a list of JSON-LD blocks, ready to inspect or validate

Run that over a list of URLs (a sitemap export or a Screaming Frog crawl) and you have a table of what’s marked up where. One caveat: this reads the HTML your server sends. If your structured data is injected with JavaScript, for example through Google Tag Manager, it won’t show up here. Google can usually render it, but many other crawlers, including most AI crawlers, don’t run JavaScript. That’s one more reason to put the markup in the HTML.

Step 2: Detect structured data opportunities with a rule-based classifier

Next, label each page. I use a deliberately simple, rule-based classifier: each rule looks at the URL, the HTML structure and the text, and every suggestion says which signal fired, so you can check it. A shortened version:

from bs4 import BeautifulSoup
import re

def detect_opportunities(html, url):
    soup = BeautifulSoup(html, "lxml")
    text = soup.get_text(" ").lower()
    path = url.lower()
    headings = [h.get_text(" ", strip=True) for h in soup.find_all(["h2", "h3"])]
    found = []

    if soup.find("meta", property="og:type", content="article"):
        found.append(("Article", "og:type=article"))
    if path.count("/") > 4:
        found.append(("BreadcrumbList", "nested URL"))
    if re.search(r"[$€£]\s?\d", text) and re.search(r"add to (cart|basket)|in stock", text):
        found.append(("Product", "price + buy wording"))
    if soup.find("iframe", src=re.compile("youtube|vimeo")):
        found.append(("VideoObject", "video embed"))
    if sum(h.endswith("?") for h in headings) >= 3:
        found.append(("FAQPage", "3+ question headings (no Google rich result since May 2026)"))
    return found

The full version in the code hub has rules for Article, Breadcrumb, Product, Course, Event, Job posting, Video, Recipe, Organization and Local business, Profile page, Review, FAQ and How-to. It tags each suggestion with its 2026 Google status, so you can sort by what’s actually worth doing. The audit command puts steps 1 and 2 together:

python sd_toolkit.py audit --urls urls.txt --out structured_data_audit.csv

You get one row per URL: the types already on the page, the opportunities detected (and why), the supported types that are missing, the retired types it found signals for, and a count of validation errors and warnings in the existing markup.

When I ran it on a handful of real sites, it surfaced exactly the kind of thing you’d want to know: course and product pages with a price, a video and a curriculum, whose only markup was FAQPage (which no longer earns anything in Google), and Product, Course and Video markup missing entirely. That’s the value of automation. Nobody spots that page by page.

Why rules and not a machine learning model? Because you’ll be generating code that goes on live pages, and you need to explain every decision. Rules are transparent and easy to fix. If your site has thousands of page types that rules can’t separate, that’s the point where a classification model earns its place.

Step 3: Generate JSON-LD from your data, not by hand

This is where most of the time savings are. Instead of typing values into a generator, you map the data you already have (a CMS export, a product feed, a spreadsheet, a database table) onto schema.org properties, and let code write the JSON-LD.

I name the columns after the schema.org properties (name, price, priceCurrency, startDate…), so the mapping stays obvious to anyone who picks it up later. For each row, the toolkit builds one connected @graph: the Organization and WebSite (shared across the site), the WebPage, a BreadcrumbList generated from the URL, and the page’s main entity. Everything links together with @id references, so search engines see one entity graph rather than a pile of disconnected snippets. (If @id is new to you, read what schema.org @id does and why your entity graph falls apart without it.)

python sd_toolkit.py generate --csv example_data/pages.csv \
    --site example_data/site.json --out-dir jsonld/

For a product row, the main entity comes out like this:

{
  "@type": "Product",
  "@id": "https://www.example.com/shop/seo-workbook/#main",
  "name": "SEO Audit Workbook",
  "description": "A printable workbook for technical SEO audits.",
  "image": "https://www.example.com/img/workbook.png",
  "sku": "WB-001",
  "brand": {"@type": "Brand", "name": "Example Academy"},
  "offers": {
    "@type": "Offer",
    "price": "29.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock",
    "url": "https://www.example.com/shop/seo-workbook/"
  },
  "mainEntityOfPage": {"@id": "https://www.example.com/shop/seo-workbook/#webpage"}
}

Two small rules make a big difference. First, empty values are dropped, so the markup only states what you actually know (no "price": ""). Second, every generated graph is validated before it’s written (step 3 of the next section), so broken rows are flagged instead of shipped. The tool writes a .json file and a ready-to-paste <script type="application/ld+json"> tag for every page.

How to upload structured data to your website

Traditional ways to add schema markup to your website

There are many ways to add structured data to your website the traditional way. Here are the most popular:

• manually inserting the JSON-LD on the webpage via the page editor (for example, a Custom HTML block in WordPress);
• code or script injection in the page header or footer, or via the functions.php file in a child theme (other options are available for specific CMSs);
• a WordPress plug-in, like Schema Pro or Schema & Structured Data for WP & AMP, or the schema features built into SEO plug-ins like Rank Math and Yoast, which already output a basic Organization, WebSite, WebPage and Article graph;
• Google Tag Manager.

Now, let’s talk about the best way to implement your structured data programmatically.

How to upload schema to your website programmatically

In terms of uploading the structured data to your webpages programmatically, this can be achieved with the help of plug-ins, custom scripts, or even Google Tag Manager. Chris Goddard has previously shared examples of the dynamic generation and insertion of article structured data using GTM, which is now the default approach of many WordPress plug-ins. This set-up builds a template for structured data, which pulls elements from the webpage HTML to complete the schema markup per the requirements, and automatically inserts it on the page once completed. (Remember the caveat from step 1: markup added with JavaScript is invisible to crawlers that don’t render it, so server-side is the safer choice when you have it.)

Uploading structured data programmatically when the elements are stored in a database, rather than directly available on a webpage, is a multi-step process. You retrieve the data from the database, structure it according to the required schema (JSON-LD, for SEO), and then embed or link this structured data to the relevant webpage. Here’s a simplified overview:

1. Access the database where your data is hosted. First, establish a connection to your database using the appropriate database drivers or APIs for your programming language of choice (for example, JDBC for Java, or psycopg for Python with PostgreSQL). Make sure you have the permissions to read the data, and use a read-only user if you can.
2. Query the database. Execute SQL queries to retrieve the data you need for the structured data snippet. This might involve selecting specific rows and columns, joining tables, or applying filters to get exactly the elements you need.
3. Format the data into the schema.org specification (if not done already). Once you have the data, format it according to the schema you’re using. This involves creating a structured format that follows the guidelines of the schema type, for example organising product data into name, offers, price and availability, following the schema.org Product specification.
4. Embed the structured data into your webpages (options below).

In Python, steps 1 to 3 are a few lines. The toolkit includes a small SQLite demo; for PostgreSQL or MySQL, only the connection line changes:

python example_data/make_demo_db.py
python sd_toolkit.py generate --sqlite example_data/demo.db \
    --query "SELECT * FROM products" --site example_data/site.json --out-dir jsonld/

There are a few ways to include this structured data on your webpage:

• Direct embedding: if you can dynamically generate webpages (using a server-side language like PHP, Python, or Node.js), you can embed the structured data directly into the HTML of the webpage before it’s sent to the browser. This is the most robust option.
• AJAX and APIs: for more dynamic websites, you can use JavaScript to fetch structured data from an API endpoint that returns it in the correct format, then inject it into the page as a JSON-LD script. It works for Google, with the rendering caveat above.
• Server-side injection: another approach is to use server-side technologies to automatically inject the structured data into webpages based on the page’s content or metadata. This can be done through template systems, server-side rendering, or web server configurations. In WordPress, for example, you can store the generated JSON-LD in a custom field (via the REST API or WP-CLI) and print it in the page head with a few lines in your theme.

Structured data testing and validation

Traditional ways to test and validate schema markup

There are many tools for page-by-page testing and validation. Here are the ones I use, updated for 2026 (two of my original favourites have since been retired):

• Google Rich Results Test (Google, web-based): checks eligibility for Google’s rich results, and renders JavaScript;
• Schema Markup Validator (schema.org, web-based): validates any schema.org markup, not just Google’s features. It replaced Google’s old Structured Data Testing Tool, which was retired in 2021;
• the Enhancements reports in Google Search Console, which show errors across all the pages of each type Google has found;
• URL Inspection in Bing Webmaster Tools, which now covers markup (Bing’s standalone Markup Validator has been retired, and so has the Ryte Structured Data Helper extension).

Here’s how to test and validate structured data implementation programmatically.

How to automate structured data testing and validation

None of the tools above has a public API for bulk testing, so you can’t run the Rich Results Test over 10,000 URLs. What you can do is automate the checks they make. I validate in three layers, from the most basic to the most useful:

1. Is it valid JSON? A missing comma or an unescaped quote breaks the whole block, and search engines will ignore all of it.
2. Is it valid schema.org? The toolkit downloads the current schema.org vocabulary (V30.1 at the time of writing) and checks every node: does the @type exist, does every property exist, is the property expected on that type, and has it been superseded? This catches typos like "@type": "Blogpost" or "colour", which generators happily let through.
3. Does it meet Google’s requirements, and does it match the page? For each type that still has a Google feature, it checks the required and recommended properties (a Product needs a name plus offers, review or aggregateRating; an Event needs name, startDate and location; an Offer needs a price…). It then checks that names, headlines and prices in the markup actually appear in the page’s visible text. Markup that doesn’t match the page breaks Google’s structured data policies, and it’s the most common problem with generated markup.

python sd_toolkit.py validate --file jsonld/shop__seo-workbook.json
python sd_toolkit.py validate --url https://www.example.com/blog/some-post/

Here’s what it reports for a deliberately broken file (shortened):

[error  ] block0.Product#1: 'colour' is not a schema.org property
[warning] block0.Product#1: 'author' isn't expected on Product (expected on CreativeWork, Rating)
[error  ] block0.Offer#1: Google requires 'price|priceSpecification' on Offer
[info   ] block1.FAQPage#1: FAQPage: no Google rich result any more. Fine to keep if it matches the page.
[error  ] block2.Blogpost#1: @type 'Blogpost' is not a schema.org type

Errors break eligibility, warnings are missed opportunities, and “info” tells you about markup that’s valid but no longer earns anything in Google.

Run the audit on a schedule (monthly, or after every template release) and you have structured data monitoring: new templates without markup, plug-in updates that break a graph, prices in the markup drifting away from prices on the page. Then spot-check one URL per template in the Rich Results Test, because that’s the final word on Google eligibility, and watch the Search Console Enhancements reports.

Where do LLMs fit in structured data automation?

It’s tempting to paste a page into ChatGPT and ask for the schema. For a one-off, fine. For automation at scale, I’d keep large language models away from the values. A model that invents a price, a date or an author on a live page is a structured data policy problem, and it’s very hard to spot by eye. Generative AI is great at being creative, not at being precise.

Where they do help: classifying ambiguous page types that your rules can’t separate, suggesting which schema.org type fits a new template, and drafting the mapping between your database columns and schema.org properties. A human checks that once, and then deterministic code does the rest. Use the model to help design the pipeline, not to fill it.

And if you’re already generating content programmatically, have a look at how to use generative AI with structured data for programmatic SEO, which comes at the same problem from the other direction.

What to look out for next…

Google will keep pruning rich result types; four rounds of retirements in three years make that clear. So build your automation around your data and your entities, not around any single rich result. When a feature disappears, you change one line in a status table rather than rebuilding your pipeline. That’s exactly how the toolkit is set up.

You can find everything in the MLforSEO code hub: the audit, generation and validation commands, sample data (CSV and a SQLite database) and a README with the current Google status of every type. For more on how structured data supports entity understanding, read how structured data reinforces entity understanding in AI search and why consistency beats completeness.

Thanks for reading, and let me know how you get on.

Related glossary terms

• Entity
• Knowledge graph
• Ontology
• Google Cloud AutoML