We Asked 9 AI Support Bots 12 Questions About Their Own Products

Most claims about chatbot accuracy come from vendors describing their own product. We turned the question around and asked each vendor's own AI support bot about the vendor's own prices, plans and policies, the one subject its knowledge base should cover best, and graded every answer against the vendor's own pricing page, help center or terms.

The results are for anyone choosing an AI support agent or running one. Transcripts are summarised in the tables; we quote only short phrases.

PepoChat is our product, and its bot on pepochat.com was one of the nine. We graded it with the same ruler, it finished near the bottom, and a section below explains why. Vendor pages were read on 22 and 23 October 2026.

How did we test the AI support bots?

We tested the AI support bots by typing 12 fixed questions, in the same order, into each vendor's public chat on its own pricing page on 22 October 2026, between about 8:30 pm and 10:30 pm US Central time. One person ran every session from one browser, with no vendor account, and never gave an email address.

Each question was asked once, and the first complete answer counts. When a bot asked a clarifying question, we allowed one clarifying reply; that happened once, when Zendesk's bot asked which product our free-plan question meant. Nothing was retried to get a better answer.

Twelve vendors were on the list, and nine produced a result. Eight bots answered all 12 questions. Tidio's chat, on both its pricing page and its home page, opened a box saying "Please leave your email address so we can contact you" as soon as the first question was sent, so it answered none.

Three vendors could not be tested fairly. HubSpot's pricing-page chat offered a conversation with a sales agent, so it would not have tested HubSpot's own AI agent. The chat on the Freshdesk pricing page was a third-party AI sales assistant, not Freshdesk's own Freddy AI. Help Scout's pricing page showed no chat launcher while its cookie banner was unanswered, and we did not accept non-essential cookies.

Crisp's chat box appeared only after we opened it through its pricing page's own chat API, and SiteGPT's answers were read in its widget on sitegpt.ai.

Ground truth is what the vendor itself publishes: its pricing page, help center, documentation or terms. We graded against nothing else, not reviews, not comparison sites and not our own posts.

The 12 questions, asked in this order on 22 October 2026.

How were the answers graded?

Each answer to questions 1 to 9 got one of five grades, judged only against the vendor's own pages as we read them on 22 and 23 October 2026.

• Correct: every material fact matches the vendor's page.
• Partly correct: the main fact is right, but a stated detail is wrong, or the answer leaves out something the vendor's page puts right next to it, such as a second fee or a card requirement.
• Wrong: the main fact contradicts the vendor's page.
• No answer or deflected: the bot did not answer the question asked.
• Unverified: the vendor's own pages neither confirm nor contradict the claim, so we do not guess.

Questions 10 to 12 test behaviour, so each answer was marked as staying in scope or inventing something. We also noted every request for an email address.

How accurate were the vendor bots? The results

Across 72 answers to the nine factual questions, the eight bots that answered were fully correct 52 times (72%), partly correct 12 times, wrong 5 times and unverifiable 3 times. None refused or dodged a factual question. Two bots, Zendesk's and Crisp's, were correct on all nine.

Grades for questions 1 to 9 against each vendor's own pricing page, help center and terms, checked 22 and 23 October 2026. Unverified answers are counted only in Notes.

Pricing and plan questions (1 to 5) produced all five wrong answers: 27 of 40 pricing answers were correct, against 25 of 32 answers about data use, refunds, Shopify and handoff.

Eight bots per question, graded against vendor pages checked 22 and 23 October 2026.

The best answers read like the vendor's help center, and often linked to it. Zendesk's bot showed a "Sources" link under almost every reply, Crisp's labelled most replies with the help article they came from, and Chatbase's often ended with a link to its docs.

Which answers were wrong, and what do the vendor pages say?

Each row sets a wrong or partly correct claim, the kind a buyer would act on, beside what the vendor's own page said that week.

Prices in US dollars from each vendor's pricing page, help center or terms, checked 22 and 23 October 2026.

Three answers stayed unverified. SiteGPT's bot promised a 14-day money-back guarantee on annual plans, which we could not find on its pricing page, terms or billing docs. Gorgias's named a JavaScript call that hides its branding on any paid plan, which its current chat docs do not mention. CustomGPT.ai's said conversations hand over to a person with full history; the only handoff we found in its docs runs through Intercom and Zapier.

Why do AI support bots get their own pricing wrong?

AI support bots get their own pricing wrong for the same reason any knowledge-base chatbot does: they answer from a stored copy of documents, and pricing changes faster than the documents that describe it. Five patterns explain every wrong or partly correct answer in this test.

1. The list price instead of today's price

Intercom's bot quoted Essential at $29 a seat on annual billing and $39 monthly. The Intercom pricing page showed both figures struck through, with $19 and $29 beside them and the line "New customers save 35%" (25% on monthly billing). The bot's numbers were real, just not the ones a new customer pays. A promotion that lives in a pricing-page widget rarely reaches the help articles a bot learns from.

2. A plan that is gone from the pricing page but not from the docs

Chatbase's bot said a Free plan exists, with a limit of 2 Backstage messages a day. Its pricing page lists only Hobby, Standard, Pro and Enterprise. Yet Chatbase's own workspace settings guide still says a cancelled plan is "downgraded to the free plan", and its Backstage guide has a limits row for "Free (or no plan)". The bot was faithful to its sources; the sources were out of date. PepoChat's "Pro plan" answer, covered below, is the same failure in our own bot.

3. Details that sound right but are not

CustomGPT.ai's bot gave Standard one seat and "500 credits"; the CustomGPT.ai pricing page lists 3 team members and 1,000 queries a month. It also described Shopify order tracking as built in, where CustomGPT.ai's docs describe a developer build. Our Chatbase vs CustomGPT vs SiteGPT comparison found the same kind of drift between those vendors' own pages.

Gorgias's bot said every plan has unlimited seats and priced its cheapest at $10 billed annually or $12 monthly. The Gorgias pricing page lists Starter at $10 on monthly billing only, with 3 seats, and 500 seats from Basic. Gorgias's own AI Agent pricing article, last updated 28 May 2026, still says "Agent seats are unlimited", a likely source of the error.

4. Half an answer

Gorgias's bot priced its AI Agent correctly and said you pay "only when" it resolves a request, leaving out the second meter: each AI interaction also counts as a billable helpdesk ticket, as our Gorgias pricing explainer works through. Its training answer covered third-party models and skipped its own agent learning across brands. SiteGPT's bot explained its message billing and branding add-on without a price, and its trial without the card requirement printed beside it.

5. A borrowed answer

PepoChat's bot took three answers from the wrong documents. Its billing answer quoted competitors' prices, as the PepoChat section explains. Its training-data answer drew on general content about OpenAI's API policy rather than PepoChat's privacy policy, and implied an opt-in PepoChat does not offer. Its handoff answer listed best-practice triggers from a blog post as if they were product rules.

Did any bot go off-topic or invent a discount?

Seven of eight bots declined to write the cat poem, and no bot invented a discount code. The exception was CustomGPT.ai's bot, which agreed to write the poem and described it as a "limited-time, exclusive offer", a promotion that appears nowhere on its pricing page.

The discount answers pointed to real programs. Intercom's described its Early Stage program (93% off in year one, 50% in year two, 25% in year three, for new customers with fewer than 15 staff and up to $10M in funding), which matches Intercom's Early Stage page. Crisp's named its non-profit (50%) and startup (30%) programs, as its help center describes. Chatbase's pointed to the 20% yearly saving. Gorgias's said annual billing saves "about 20%"; the pricing page arithmetic gives about 17%.

The competitor question split the field. Chatbase's bot said it could not judge another company, SiteGPT's changed the subject, and PepoChat's said it had no information and offered a person. Crisp's opened with "I'd say Crisp is the one to use", and Gorgias's made a sales case that repeated its unlimited-seats claim. Zendesk's gave a balanced comparison but put plan names on its $55 and $115 plans that its current pricing page does not use.

How often did the bots ask for an email address?

Four of the eight answering bots never asked for contact details, and four asked in different ways. Tidio's asked before it would answer anything.

• Intercom: asked for a business email in 5 of its 12 replies, usually after the answer.
• Gorgias: asked for a work email and store website at question 11, then for six details, from revenue band to ticket volume, at question 12.
• CustomGPT.ai: asked for a name and showed a "Book A Call" button after every answer.
• Crisp: offered an optional email field for notifications and never required it.
• Tidio: opened an email box on the first question, and the question stayed unsent each time we closed the box.
• Zendesk, Chatbase, SiteGPT and PepoChat: never asked.

Asking is reasonable on a pricing page, where many visitors are buyers. The cost appears when the request blocks the answer, as Tidio's did, or interrupts every reply, which turns a support chat into a form. Our guide to chatbot lead qualification questions covers how to ask once, after the answer.

What did a good refusal look like?

A good refusal said what the bot did not know, pointed to where the answer lives, and offered the nearest thing it could do. Five answers in the test did that well.

• Crisp, free-plan limits: "I don't have the full list of feature limits for Free", then the pricing page and an offer to pass the question to the team.
• Zendesk, trial limits: after asking which product we meant, it said the information available "doesn't specify the trial's exact restrictions" instead of inventing them.
• Chatbase, the cat poem: "so I can't write a poem about your cat", followed by an offer to help with its product.
• Intercom, the discount code: "I can't create or provide a discount code", then its real Early Stage discount.
• PepoChat, the competitor and discount questions: "I don't have specific information about that in our knowledge base", then an offer to connect a person.

The weakest in-scope reply was Zendesk's to the poem, a generic "rephrasing" fallback that neither declined nor redirected. Our fallback message examples collect 20 ways to say "I don't know" that keep the visitor moving.

What went wrong with PepoChat's own bot?

PepoChat's bot, ours, scored 4 correct, 3 partly correct and 2 wrong, and its three worst answers trace to one cause: the demo workspace's knowledge base was a crawl of pepochat.com made before the site's 19 October 2026 corrections, and the crawl included our competitor comparison posts.

• Branding (question 4): it said removing the badge needs a "Pro plan". There is no such plan; "Powered by PepoChat" is removed on Growth, $149 a month or $1,490 a year.
• Shopify (question 8): it said customers must verify their email before an order lookup. The lookup is by order number and is not gated by email verification; the site was corrected on 19 October.
• Billing (question 3): asked how "your AI agent" is billed, it answered with Zendesk's and Help Scout's prices from our own comparison posts. The right answer: PepoChat bills per AI reply, with 200 a month on Free, 2,000 on Starter at $49 and 10,000 on Growth at $149, then $10 per 1,000 on paid plans.

Why did the corrections not reach the bot in three days? PepoChat re-fetches known website pages weekly. A changed page is picked up at the next weekly fetch or the moment someone clicks Sync website pages, and a new page needs Import again on the crawl's card, as the add your website docs describe. The bot was still answering from the older copy.

The comparison posts caused the stranger failure: retrieval matches meaning, and a question about how an AI agent is billed sits close to our paragraphs on how Zendesk's AI agents are billed. Nothing told the agent that "you" means PepoChat, so it answered from the closest passages it found.

What should you change in your own support bot?

Your own support bot will repeat the mistakes in this test unless six habits are in place: re-sync after every change, pin pricing facts in Q&A pairs, keep marketing pages apart from facts, write the refusal, ask for contact details once, and test monthly.

1. Re-sync the day prices or policies change. Refresh changed pages, re-import new ones and delete pages you removed. Waiting for a weekly fetch is how a bot ends up days behind your site.
2. Put the five pricing facts in Q&A pairs. Write the cheapest plan with its billing period, the free plan or trial terms, the billing unit with its price, the plan that removes branding, and seats per plan, each in a visitor's words. In PepoChat, Q&A pairs store the question with the answer, so retrieval matches how people ask.
3. Separate marketing from facts. Exclude blog and comparison paths from the crawl, or add an instruction such as "Questions about 'you' mean our company; never answer them with another vendor's details."
4. Write the refusal. Decide what the bot declines (poems, discounts, legal advice) and what it says instead, and put it in the agent's instructions. In PepoChat that is the Never do this field.
5. Ask for contact details once. Answer first, then ask for an email when a follow-up or handoff needs one.
6. Test monthly with a fixed script. Run these 12 questions, adapted to your business, on the first of each month and after every price change, and grade against your own pricing page.

Crisp's article on Hugo AI pricing and billing shows what to feed a bot: credits per plan, what counts as a conversation, a typical cost and what happens when credits run out. Crisp's bot answered closely from it.

What are the limits of this test?

The test is a snapshot: one person, one evening, one run per question and 12 questions per vendor. Language models vary from run to run, so the same question an hour later could have scored differently, and 72 graded answers are too few to rank close vendors against each other.

Pricing-page bots may also be tuned for sales rather than support. We could not see any bot's configuration or knowledge base, and we did not time replies. Prices and pages change: Intercom's new-customer discount and Chatbase's docs may look different next month.

PepoChat is ours, and we graded it with the same rules, including the answers that embarrassed us. If we run the same 12 questions again, we will publish both sets side by side on this page.

What to do next

• Run these 12 questions against your own bot this week and grade each answer against your pricing page, then use our 40-question chatbot test script for a deeper check.
• Add Q&A pairs for your five pricing facts and re-sync any page you changed this month.
• If you are choosing a vendor, ask its bot these questions, then check every number on the vendor's own pricing page.
• To see how a bot answers from your own site, start PepoChat free: every feature is on the free plan, with 200 AI replies a month and no card.