Comparing NLP APIs for Entity and Sentiment Analysis (vs Generative AI)

“Just use ChatGPT for it.” I hear this constantly about entity analysis, and I understand why: it is accessible and easy. But entity analysis sits at the centre of so many SEO projects, from understanding your content at scale to the queries you rank for and the reviews customers leave, that getting it right genuinely matters. So I tested it. I ran one text through three purpose-built NLP APIs (Google Cloud Natural Language, Amazon Comprehend and IBM Watson NLU) and gave the same text and the same prompt to four generative AI chatbots (ChatGPT, Gemini, DeepSeek R1 and Alibaba’s Qwen).

This post is the full write-up of that benchmark. It started as the comparison lesson in the entity module of my Introduction to Machine Learning for SEO course and as a short summary in MLforSEO Newsletter #002. Here I set out the research question, the exact setup, the test text and prompt, the results for every model, an error analysis of the raw outputs, the limitations, and how to reproduce it yourself.

On this page

1. Research Question: Purpose-Built NLP APIs or Generative AI for Entity Extraction?
2. What the Research Already Says About LLMs and Named Entity Recognition
3. The Contenders: Purpose-Built NLP APIs vs Generative AI
4. Method: How I Set Up the Comparison
5. Results: Entity Counts for Every Model
6. Error Analysis: Six Ways Entity Extraction Went Wrong
7. Repeatability: Same Prompt, Different Answer
8. Why Generative AI Falls Short for Entity Extraction
9. What About Sentiment Analysis?
10. Cost and Practicality at Scale
11. Free vs Paid: Cloud APIs vs Open-Source Libraries
12. Limitations of This Comparison
13. Practical Recommendations
14. How to Reproduce This Experiment
15. Why Reliable Entity Data Matters for AI Search
16. Where to Take Entity and Sentiment Analysis Next

Research Question: Purpose-Built NLP APIs or Generative AI for Entity Extraction?

The question I set out to answer is a practical one: for extracting entities from marketing text, do pre-trained, task-specific NLP APIs give better results than general-purpose generative AI chatbots used with a simple prompt? “Better” here means four things an SEO actually cares about: how many of the real entities a tool finds, how well it types them (person, organisation, location and so on), how much additional data it returns for analysis, and whether you get the same answer twice.

I ran the test in early 2025, shortly after DeepSeek R1 and the Qwen 2.5 models were released, to check whether my long-standing recommendation (a task-specific API, not a chatbot, for entity work) still held.

What the Research Already Says About LLMs and Named Entity Recognition

Entity extraction is known in the research literature as named entity recognition (NER). The standard way to evaluate it was set by shared tasks such as CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003), which used four entity types (persons, organisations, locations and miscellaneous names) and scored systems with precision, recall and their harmonic mean, the F1 score. Precision is the share of the entities a system returns that are correct; recall is the share of the true entities it finds. In CoNLL scoring an entity only counts as correct if it is an exact match of the annotated span. I come back to these definitions below, because they explain why a raw “number of entities found” can mislead.

Several studies have compared large language models with supervised NER systems, and the pattern is consistent:

• Qin et al. (2023), “Is ChatGPT a General-Purpose Natural Language Processing Task Solver?”, found that ChatGPT does well on many reasoning tasks but still struggles with specific tasks such as sequence tagging, the family NER belongs to.
• Wang et al. (2023), “GPT-NER”, open by noting that LLM performance on NER is still significantly below supervised baselines, because NER is a sequence-labelling task while LLMs are text generators. They close much of the gap only with a purpose-designed prompting and self-verification method, not with a simple instruction.
• Han et al. (2023, revised 2024), “An Empirical Study on Information Extraction using Large Language Models”, report a visible performance gap between GPT-4 and state-of-the-art information extraction methods.
• Atil et al. (2024, revised 2025), “Non-Determinism of ‘Deterministic’ LLM Settings”, show that LLM outputs vary across repeated runs even with settings meant to be deterministic, with accuracy swinging by up to 15% between runs.

My experiment is far smaller, but it asks the same question where most marketers meet it: a chatbot window and a pasted paragraph.

The Contenders: Purpose-Built NLP APIs vs Generative AI

On one side, pre-trained, task-specific NLP APIs from the tech giants: Google Cloud Natural Language API, Amazon Comprehend and IBM Watson NLU (you could add Microsoft Azure to this group too). On the other, the generative AI camp: ChatGPT (OpenAI’s GPT models), Google Gemini, DeepSeek R1 and Alibaba Qwen, a deliberate mix of the established players and the newer arrivals.

Update (October 2026): IBM deprecated Watson Natural Language Understanding on 31 July 2026, and the service will be removed from IBM Cloud on 31 January 2028 (IBM release notes). IBM asks existing customers to migrate before then, with Watson Natural Language Processing in watsonx.ai as its suggested route. The IBM results below describe the API as I tested it.

The first thing to understand is that these NLP APIs are not single-trick tools. Each bundles a suite of modules for different natural-language tasks, and they overlap in useful and slightly different ways.

Google Cloud Natural Language API gives you entity identification (it inspects text for known entities and returns rich information on each), entity sentiment (the sentiment attached to a specific entity, so a review that mentions “price” negatively gets a negative score on that entity), document-level sentiment and, uniquely among the three, content moderation, plus text classification and syntax analysis. Amazon Comprehend covers entities, entity sentiment and document sentiment, and adds key-phrase (keyword) extraction, language detection, PII detection and syntax analysis. IBM Watson NLU is the most feature-rich for our purposes: on top of entities, entity sentiment and document sentiment, it does entity-relationship mapping (its “relations” module), emotion detection (joy, anger, sadness and more), keywords, concepts, classification, metadata and semantic roles.

Method: How I Set Up the Comparison

Test text

I used a single English text of 221 words (1,490 characters, ten sentences): the professional bio I use on MLforSEO. I chose it because it is dense with the kinds of entities SEOs work with every day: a person, her own companies, a university, and a list of client brands, mixed with industry concepts such as SEO, machine learning, B2B and SaaS. It also has two deliberately awkward cases: an organisation name that contains a country (“Women in Marketing – Bulgaria”) and a brand name that contains a number (“Neo4j”).

Prompt for the generative models

Every chatbot received exactly the same instruction, followed by the text:

Please extract the entities you recognize from this text. Organise them into entity type categories. Format your response to be in a tabular format with column 1 numbers, column 2 entity type and column 3 entity name

This is a zero-shot prompt (no examples, no allowed entity types), deliberately: it is how most people use a chatbot for this task.

Models, versions and access

All tools used their default settings; nothing was fine-tuned or customised. The chatbots were used through their consumer interfaces, so temperature and other generation settings were whatever each product applies by default.

Procedure and measures

I collected each tool’s raw output into one Google Sheet, one tab per tool, and counted three things for each: the total number of entities returned (rows), the number of unique entity names, and the number of entity types (categories) used. I also re-ran Qwen with the same prompt to check whether a chatbot gives the same answer twice. Those counts are what I presented in the course lesson and the newsletter.

For this write-up I went back to the raw outputs and added two checks that the counts alone cannot show. First, a recall check on core named entities: the text contains 13 unambiguous proper-noun entities (Lazarina Stoy, MLforSEO, MLforSEO Academy, Women in Marketing – Bulgaria, University of Strathclyde, AWS, and the seven client brands Compass Real Estate, Skims, Skyscanner, Extreme Networks, Neo4j, Chemical Guys and Perkbox), and I checked which tools returned each one as a full, exact entity. Second, an error analysis: I read every returned row and noted entities that are not in the text, names that were split or merged, wrong types and wrong knowledge-base links. This is a single-annotator check on one text, not a full precision/recall evaluation, and I treat it as such in the limitations.

Results: Entity Counts for Every Model

Here are the headline numbers as I reported them in the lesson and newsletter:

Table 1. Counts as originally published in the course lesson slides and MLforSEO Newsletter #002 (February 2025).

When I re-checked the spreadsheet for this write-up, I found that most of the summary formulas had included the column header in their count, which inflated several cells by one. The ranking does not change, but for transparency here are the corrected counts, recounted directly from the raw outputs, alongside the core named-entity check:

Table 2. Recount of the raw outputs (October 2026). “Unique” counts distinct entity names. The core named-entity check counts a hit only for the full, correct name.

Read these two tables together, because they tell slightly different stories. On volume, Google Cloud is the clear winner and DeepSeek R1 is the strongest generative model, which is what I concluded in the lesson. On finding the entities that matter most, the people and organisations, the picture is much flatter: four tools found all 13, including ChatGPT with the smallest output. The difference between the tools is mostly in what else they return, how they label it, and what they get wrong.

Google Cloud Natural Language API

Google returned 70 entity rows, 59 of them unique, across six types (OTHER, ORGANIZATION, PERSON, LOCATION, EVENT and WORK OF ART). It found all 13 core entities. The volume comes from how broadly Google defines an entity. Its documentation describes entity mentions as either proper names or common nouns, and the output reflects that: alongside the brands, it returned industry concepts (SEO, machine learning, data science, SaaS, B2B) and a long tail of common nouns such as “teams”, “positioning”, “passions” and “more”. Of the 59 unique entities, by my reading 16 are proper-noun names, 12 are industry concepts an SEO would want, and 31 are generic common nouns. Some generic items were also oddly typed, for example “community” and “women” as PERSON. The noise is easy to filter using the salience score and mention type, and in exchange you get the richest data in the test.

Amazon Comprehend

Comprehend returned 21 rows, 15 unique, across five types, and found 12 of the 13 core entities. It missed “Women in Marketing – Bulgaria” as an organisation, returning “Bulgaria” as a location and “Bulgarian” as OTHER instead. It typed MLforSEO as a TITLE twice, a type Amazon reserves for names of creative works such as books and films. Every entity comes with a confidence score, most of them above 0.9.

The lower count needs context. Concepts that Google labels as OTHER entities, such as big tech, machine learning and marketing, Comprehend treats as key phrases, which is a separate module. Running its key-phrase model on the same text added more than 50 further phrases. Part of the gap is how each provider organises its outputs, not raw capability.

IBM Watson NLU

IBM returned 16 rows, 15 unique, across five types. It found only 7 of the 13 core entities in full and missed five of the seven client brands. It was also the tool with the most visible errors: it split “Women in Marketing – Bulgaria” into a job title (“Founder of Women”) and a location (“Marketing – Bulgaria”), typed “Lazarina” as an Organization as well as a Person, returned “B2B, SaaS” as a single organisation, and extracted the number 4 (from “Neo4j”). Where IBM shines is the extra data: every entity carries a relevance score, a confidence score, a mention count and entity-level sentiment, and four entities were linked to DBpedia. Three of those links were correct (University of Strathclyde, Search engine optimization and Extreme Networks). One was wrong in an instructive way: AWS was linked to the American Welding Society.

ChatGPT (o1)

ChatGPT returned 14 entities in three types (Person, Organization, Location) and found all 13 core entities plus Bulgaria. Every item it returned appears verbatim in the text. It was the most conservative tool: precise on named entities, but with no concepts and no supporting data to analyse beyond the list.

DeepSeek R1

DeepSeek returned 38 rows (37 unique, because it listed the University of Strathclyde under two types) across eight types, including Professional Roles, Concepts/Topics, Products/Services and Events. It found all 13 core entities. It was the most generous of the chatbots, and that generosity is also its main risk. Ten of its 37 entities are not in the text as written. Some are harmless expansions (“SEO (Search Engine Optimization)”, “AWS (Amazon Web Services)”), but others are inferences the text does not support: it listed “SEO Strategist” and “Data Scientist” as roles, although the text only says I specialise in SEO and data science.

Qwen2.5-Plus

Qwen returned 25 entities across six types and found all 13 core entities. Its typing was inconsistent: organisations appear as Organization, Company or University, and MLforSEO Academy as a Product. Twelve of its entities were industry concepts grouped as “Field”. When I ran the same prompt on the same text a second time, Qwen returned 24 entities with a different ordering and grouping. That one re-run is the clearest illustration in this test of the repeatability problem.

Gemini

Gemini returned nothing. Both Gemini Advanced models I tried, 1.5 Pro with Deep Research and 2.0 Experimental Advanced, replied that they did not have enough information about that person to help. Gemini appears to have read a text that starts with a person’s name as a request for information about that person, rather than as a text to process. It is a refusal, not an extraction error, but the practical result for a workflow is the same: no data.

What else each tool gives you beyond the entity list

Table 3. Output fields available from each tool in this test, based on the outputs collected and each provider’s documentation.

Error Analysis: Six Ways Entity Extraction Went Wrong

Reading the raw outputs row by row turned up six kinds of error. Every tool that returned output made at least one.

1. Refusal. Gemini declined the task on both models, treating the text as a person lookup.
2. Inferred or reworded entities. DeepSeek returned ten entities not found verbatim in the text, including roles the text never assigns. Qwen expanded two acronyms. In a workflow that matches entities back to pages, an entity that is not in the text is a hallucination, however plausible it sounds.
3. Split names. IBM broke “Women in Marketing – Bulgaria” into two wrong entities, and Comprehend reduced it to a location. The country inside the name tripped both.
4. Wrong type. IBM typed my first name as an Organization and “B2B, SaaS” as one organisation; Comprehend typed MLforSEO as a TITLE; Google typed “community” as a PERSON.
5. Wrong knowledge-base link. IBM linked AWS to the American Welding Society. Entity linking is powerful but needs checking, especially for short acronyms.
6. Noise. Google returned dozens of generic common nouns. They are easy to filter, but they inflate raw counts, which is why volume alone is not a fair score.

The two most important errors for SEO work point in opposite directions. The APIs make mistakes you can see and filter, because every entity comes with a type, a score and a position in the text. The chatbots make mistakes that look like correct answers: a tidy table with a plausible role or an expanded name in it, and nothing to tell you which rows to trust.

Repeatability: Same Prompt, Different Answer

A task-specific API is a fixed model: send it the same text and you get the same entities back, which means anyone can reproduce your analysis. A chatbot samples its answer. In this test, re-running Qwen on the identical prompt and text produced 24 entities instead of 25, grouped differently. That matches what Atil et al. (2024) found at much larger scale: outputs from LLMs vary between runs even with settings designed to make them deterministic.

For a consultant this matters: if a client re-runs your analysis and gets different numbers, you are defending a result nobody can replicate.

Why Generative AI Falls Short for Entity Extraction

Putting the results together, three things separate the two camps. Scalability: an NLP API processes your whole site, keyword universe or internal-link audit with a single line of code, while a chatbot handles one text at a time. Richer data: only the APIs returned fixed entity types, salience or relevance, confidence, mention counts and knowledge-base links. Repeatability: an API returns the same result every run; a chatbot does not. Chatbots can also hallucinate entities that are not in the text, invent their own category names each time, and cost more while running slower at scale. The root cause is simple: there is no database of entities inside a GPT that it matches your text against. There is one behind Google Cloud, Amazon and IBM, built and maintained over years. Generative models are trained to generate plausible text, not to reliably extract and classify structured facts, and the research above shows that closing the gap takes specialised prompting methods, not a one-line instruction. Generative AI is superb at content transformation (summarising, rewriting, question-answering), which I cover in the Introduction to Machine Learning for SEO course; it is just the wrong instrument for structured text analysis.

What About Sentiment Analysis?

Sentiment is the other half of semantic analysis, so I also looked at how the same three APIs compare there. I did not run this part myself; I rely on an independent experiment by Reinaldo Bergamaschi (2023), who manually labelled about 6,000 social media comments and ran them through the default sentiment models of all three providers, with no training.

Table 4. Sentiment results reported by Bergamaschi (2023), as summarised in my course lesson.

Amazon and Google did well on positive and negative samples, with good precision and recall, while IBM was more precise on neutral samples, though with lower recall. Two numbers are worth separating. Each API on its own matched the human label roughly 67–72% of the time. But all three APIs agreed with each other and with the human on only 43.3% of items. An independent academic study found the same pattern: Jung, Salminen and Jansen (2022) report that four commercial sentiment tools (Azure, IBM, Google and AWS) assigned the same sentiment to less than half (48.1%) of the content they analysed.

The lesson is not that these APIs are bad; they are excellent at scale. It is that you still need a human review layer before you present or act on the results, and you should test a few APIs on your own data before committing. Generative AI for sentiment is impractical out of the box: zero-shot, one-shot or few-shot prompting and a low temperature setting help, but the results stay questionable compared with a purpose-built API. (Google’s Vertex AI, which became part of Gemini Enterprise Agent Platform in April 2026, is one route if you are set on the generative path.)

Cost and Practicality at Scale

Cost is often the reason people reach for a chatbot they already pay for, so here is what the three APIs charge for entity analysis, from each provider’s official pricing page. Prices change, so check the live pages before you budget.

Table 5. Entity analysis pricing as listed on the official pricing pages, checked October 2026.

To put that in perspective, the 1,490-character test text in this experiment is two Google units. A content audit of a thousand pages of similar length sits comfortably inside Google’s free monthly allowance. One version note for Google users: the newer v2 of the Natural Language API, per its REST reference, does not include the entity sentiment and syntax methods that v1 offers, so check which version a template or script calls if you rely on entity-level sentiment.

Free vs Paid: Cloud APIs vs Open-Source Libraries

You do not have to use a paid cloud API. Open-source options like spaCy, NLTK, Stanford NLP and Hugging Face transformers (including BERT and DistilBERT) support entity recognition too, though spaCy and NLTK are more often used for the text-processing steps around it, like tokenisation and vectorisation. I did not include them in this benchmark, which focuses on the commercial APIs and chatbots most marketers have access to.

Cloud solutions win on plug-and-play usage, a broad suite of features in one API, automatic scaling and easy integration, especially if your data already lives in BigQuery. Open-source and local solutions shine when you need full data control for regulated projects, when cost at very high volume is a concern, or when you want to fine-tune for a specialised domain. Run a cost test on a cloud API before ruling it out: the price for large volumes is often surprisingly small relative to the time you save.

Limitations of This Comparison

This is a small, practical benchmark, and its results should be read with these limits in mind:

• One text, one genre, one language. The test used a single 221-word English bio. Product pages, reviews, queries or other languages could rank the tools differently.
• One run per tool (two for Qwen). With a single run I cannot report variance for the chatbots, only show that it exists.
• No independent gold standard. The core named-entity check and error analysis are my own reading of the outputs, by a single annotator, and cover the 13 unambiguous proper-noun entities rather than a full annotation of every concept in the text. I did not compute formal precision, recall or F1.
• Counts reward broad definitions. Tools that return common nouns and concepts (Google) or that put concepts in a separate module (Comprehend key phrases, IBM keywords and concepts) are not measured like-for-like by entity counts.
• Different access routes. The APIs were called through a Sheets template, a console demo and a notebook; the chatbots through their consumer web interfaces with default settings. Results via each chatbot’s API, with a fixed temperature or a structured-output schema, could differ.
• Slightly different input formatting. The chatbots received the text with paragraph breaks inside the prompt, and that version contains one sentence fragment that is not in the API version. The entities in both versions are the same.
• The text is about a real, named person (me). That is what triggered Gemini’s refusal, and models that have seen my bio online might be helped by prior knowledge.
• Models change. The chatbot versions tested in early 2025 have since been updated or replaced, and the APIs evolve too (Google’s v2 API and IBM’s updated entity type system are examples). The results describe those versions at that time.
• Counting errors. As noted above, the summary formulas in my original sheet counted the header row; Table 2 corrects this.

Practical Recommendations

• For entity work at scale, use a task-specific API. Google Cloud’s Natural Language API is my default for SEO: broadest coverage, salience, Knowledge Graph IDs and an easy no-code route.
• Filter, do not just count. Use salience, confidence and type to drop generic nouns before you analyse.
• Check knowledge-base links on short names and acronyms. The AWS-to-American Welding Society link is the kind of error that slips into a report unnoticed.
• Test two APIs on your own data first, and include their key-phrase or concept modules, because providers draw the line between entity and keyword differently.
• If you must use a chatbot, use it for one-off lists of named entities, give it a fixed list of entity types, ask it to return only text that appears verbatim, and verify the output. DeepSeek R1 was the strongest generative option in this test, but I still would not build a workflow on it.
• Keep a human review step for sentiment, and do not treat any single API’s sentiment label as ground truth.

How to Reproduce This Experiment

Everything you need to repeat or extend the test is available:

• Test text, prompt and every tool’s raw output: the results Google Sheet, one tab per tool.
• IBM Watson NLU code: the free Entity Analysis & Relationship Mapping with IBM Watson NLU Colab notebook, which extracts entities with sentiment, exports them to CSV, and maps and visualises entity relationships.
• Google Cloud Natural Language: the free no-code Entity Extraction with Google’s Natural Language API in Google Sheets template.
• Amazon Comprehend: the Amazon Comprehend console demo in your AWS account (find Comprehend from the Services search bar), which lets you analyse a text and download the results as CSV.
• Module-by-module comparison: the free Entity & Sentiment Analysis APIs: Comparative Review.
• The full walkthrough: the video lesson, slides and supporting resources are in the Entity Extraction and Analysis module of Introduction to Machine Learning for SEO.

The most useful extension would be a set of texts from different genres, several runs per chatbot, and a full annotation so precision, recall and F1 can be computed properly. I also compared GPT-4 with Google Cloud on a wider set of tasks in my earlier piece for Search Engine Land, GPT-4 vs. Google Cloud: Performance comparison on 9 SEO tasks.

Why Reliable Entity Data Matters for AI Search

There is a bigger reason to care about doing entity extraction properly than tidy data. Google has organised search around entities since it introduced the Knowledge Graph in 2012 (“things, not strings”), and AI search systems build answers out of facts, which are entities and their relationships. When you analyse your content and your SERPs with the same class of purpose-built model, you are auditing your brand the way the machines read it: which entities you are associated with, how confidently, and with what sentiment. In this test, only the APIs returned knowledge-base identifiers, which is what lets you check whether a mention resolves to the right real-world thing (and catch it when it does not, as with AWS). A generative chatbot cannot give you that picture reliably, because it has no entity database to match against. I go deeper into how AI search platforms use entity recognition and linking in my guide for iPullRank, How AI Search Platforms Leverage Entity Recognition.

Where to Take Entity and Sentiment Analysis Next

This post makes the case for purpose-built NLP APIs. To put one to work, start with the free Entity Extraction with Google’s Natural Language API guide, and read what entity extraction is and where it fits in SEO for the concepts. Then see it all applied in the semantic analysis of customer reviews walkthrough, which uses Google Cloud and IBM Watson together. The full Entity Extraction and Analysis module inside Introduction to Machine Learning for SEO takes each API hands-on, module by module.

Related glossary terms

• Entity extraction (NER)
• Entity sentiment analysis
• Entity salience score
• Sentiment analysis
• Knowledge Graph
• IBM Watson NLU
• BERT

References

• Atil, B. et al. (2024, rev. 2025). Non-Determinism of “Deterministic” LLM Settings. arXiv:2408.04667.
• Bergamaschi, R. (2023). Sentiment Analysis: Comparing Results of AWS, IBM and Google. Medium, 10 April 2023.
• Han, R. et al. (2023, rev. 2024). An Empirical Study on Information Extraction using Large Language Models. arXiv:2305.14450.
• Jung, S.G., Salminen, J. and Jansen, B.J. (2022). Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment. ACM EICS 2022.
• Qin, C. et al. (2023). Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv:2302.06476.
• Singhal, A. (2012). Introducing the Knowledge Graph: things, not strings. Google blog.
• Stoy, L. (2025). Generative AIs tested against custom-trained NLP APIs by Google, Amazon, and IBM on Entity Extraction. MLforSEO Newsletter #002, 17 February 2025.
• Stoy, L. (2025). How AI Search Platforms Leverage Entity Recognition. iPullRank.
• Tjong Kim Sang, E.F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. CoNLL 2003.
• Wang, S. et al. (2023). GPT-NER: Named Entity Recognition via Large Language Models. arXiv:2304.10428.
• Provider documentation: Google Cloud, Analyzing entities; Amazon Comprehend, Entities; IBM Watson NLU, Entity type systems.