Meta Description Generator in Python: BERT vs Embeddings vs LLM, with Guardrails

Hundreds of pages with missing or copied meta descriptions is a common audit finding, and writing them all by hand rarely happens. These three free Colab notebooks generate a description for every page in three different ways, flag anything a human should check, and save a CSV ready for review. It started as my 2021 BERT tutorial and now compares that method with the modern options.

The three methods

• Notebook 01: BERT extractive. The 2021 method with its bugs fixed. It picks one existing sentence of the right length, so it can’t invent facts. Free and local.
• Notebook 02: embeddings and a small open LLM. (a) Picks the sentence closest to the page and its top query using sentence-transformers; (b) Qwen3-1.7B writes a new description from the page text. Free and local; a GPU runtime is recommended for (b).
• Notebook 03: Claude with guardrails. A grounded prompt via the Anthropic API, with a cost estimate and a dry-run mode first. Paid per token.

Who it’s for

SEOs and content operations teams with large sites, migrations or new templates that need descriptions at scale.

How to use it

1. Prepare inputdata.csv with a URL and Content column (for example from a Screaming Frog custom extraction), plus optional Title and Top query columns. Or switch on scraping and list your URLs in urls.txt.
2. Open notebook 02 in Colab (switch to a GPU runtime for the LLM step), upload the CSV and run all cells.
3. Compare the extractive and generated versions side by side.
4. Read every flagged row, pick or blend the version you prefer, and only then import.
5. Want a hosted model instead? Open notebook 03, add ANTHROPIC_API_KEY in Colab Secrets, and check the cost estimate in dry-run mode before generating anything.

The checks

• SHORT / LONG: outside 70 to 155 characters.
• NUMBER NOT IN PAGE / NAME NOT IN PAGE: the most common hallucinations.
• TOP QUERY MISSING and DUPLICATE, plus a similarity score between the description and the page.

Limits

• The checks flag problems; they don’t prove a description is good. None of the methods knows your brand voice or your offer.
• Extractive output is accurate but often flat. Generated output reads better but can drift, which is what the flags are for.
• First runs download large models: about 1.3 GB for BERT and 3.4 GB for Qwen3.

The full tutorial is How to Automatically Generate Meta Descriptions with Python, and content transformation tasks like this are covered in the Introduction to Machine Learning for SEO course.