AI Chatbot Vendor Evaluation Checklist: 40 Questions to Ask
40 questions to ask AI chatbot vendors in eight groups, with why each matters, what a good answer sounds like, a scoring sheet, red flags and a trial plan.
Short answer
An AI chatbot vendor evaluation should cover 40 questions in eight groups: grounding and accuracy, human handoff, knowledge sources, actions and permissions, pricing unit and caps, data and security, widget performance and accessibility, and testing, reporting and exit. As of October 2026, anchor the security questions in OWASP's 2025 Top 10 for LLM Applications, NIST's AI Risk Management Framework and Article 50 of the EU AI Act, demand written answers, and verify them in a two-week trial.
Most AI chatbot demos look alike: a fast, friendly answer to a question the vendor picked. The differences that decide whether a bot helps your support team or embarrasses it show up later, when your content has no answer, when a customer asks for a person at midnight, or when a script sends 50,000 messages and the invoice arrives.
This checklist is for support leads, founders and anyone writing a request for proposal (RFP) for an AI support agent. It gives 40 questions in eight groups of five, each with the reason it matters and what a good answer sounds like, then a scoring sheet, the red flags to watch for and a two-week trial plan. It works on any shortlist, PepoChat included.
PepoChat is our product; its answers in the table near the end were checked in its code and documentation on 22 October 2026, limits included. The OWASP, NIST and EU sources and the vendor page quoted below were read on their official sites the same day.
What should you ask an AI chatbot vendor before you buy?
Ask an AI chatbot vendor 40 questions in eight groups, from grounding and handoff to pricing, data and exit, and accept only written answers that give numbers, limits and dates you can check in a trial.
A good answer is specific, testable and dated: it names a number, a setting or a document, and you could prove it wrong within a week. "Yes, we support websites" is not an answer. "Yes, up to 200 pages and three levels deep, re-fetched weekly, with a button to refresh now" is.
Each table below has four columns. The question is what you send. "Why it matters" is the reason you can paste into an RFP when a vendor asks why you need to know. "What a good answer sounds like" is what to listen for, including the honest "no, and here is the workaround" that beats a vague yes.
How do you run an AI chatbot vendor evaluation?
Run an AI chatbot vendor evaluation in seven steps over about four weeks: collect real questions, shortlist three vendors, send the 40 questions in writing, score the answers, trial two finalists for two weeks, check the contract, then decide.
- Collect 40 real questions from your inbox, email and call notes, including the awkward ones: refunds, angry customers, questions your website does not answer. They become your trial test set.
- Shortlist three vendors. More turns the evaluation into a reading exercise; fewer leaves nothing to compare.
- Send the 40 questions in writing with a five-working-day deadline, and ask for a documentation link with each answer. A written answer can go into the contract; a sales call cannot.
- Score the answers before any demo, using the sheet below, so a polished demo does not anchor your judgement.
- Trial the top two for two weeks on your own content, following the trial plan further down.
- Read the contract against the answers. The billing unit, overage, renewal, notice period, export and deletion terms should match what the vendor wrote.
- Decide, and write down why. That note is what you hold the vendor to at renewal.
Involve three people: the support lead who owns the questions, someone technical who can read API documentation and run a page-speed test, and whoever signs the budget. If you handle personal data from EU residents, add whoever owns privacy.
Questions to ask about grounding and accuracy (1 to 5)
Grounding questions test whether an AI chatbot answers only from your own content and admits when it cannot, which matters because a confident wrong answer about a price or a policy can be held against your business.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 1 | Does the bot answer only from our content, and what does it say when the answer is missing? | A bot that fills gaps from general knowledge invents policies, and customers hold you to them | "Only from your knowledge base. When nothing relevant is found, it says so and offers a person. Try it in the trial." |
| 2 | Can we see which sources a reply drew on? | Tracing a wrong answer to the page behind it turns an hour of guessing into a five-minute fix | "Each reply lists or logs the passages it used," or a plain "Not shown; here is how you debug instead" |
| 3 | Which rules can our instructions change, and which are fixed? | Tone and topics should be yours; honesty and handoff should not depend on how a prompt is worded | A list of base rules your instructions cannot switch off, and the settings that are yours |
| 4 | How do you define and measure accuracy or resolution, and can we rerun the test? | Vendor figures come from the vendor's own questions, definitions and dates | The definition, the test set and its date, plus a free way to run your own questions |
| 5 | What happens when two sources disagree or a page changes? | Stale and conflicting content cause most wrong answers after launch | A re-sync schedule, a refresh-now button, and replace-on-upload so old copies do not linger |
OWASP's entry on misinformation (LLM09) notes that Air Canada was successfully sued after its chatbot gave travellers wrong information, and the first mitigation it lists is retrieval-augmented generation: grounding replies in trusted content. Question 1 asks whether the vendor does exactly that.
In the trial, ask five questions your content does not answer. A grounded bot says it does not know and offers a person; a weak one guesses politely, and the guess sounds just as confident as a right answer.
Questions to ask about human handoff (6 to 10)
Handoff questions test what happens when the bot cannot help, which is when customers judge you most: a bot that loops, promises a person at 2 a.m. or loses the conversation turns a simple question into a complaint.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 6 | What triggers a handoff: a request for a person, no answer, frustration, an error? | A bot that keeps answering instead of escalating creates your angriest customers | Every trigger listed, and "talk to a human" honoured at once, without a quiz first |
| 7 | Where does a handed-off conversation land, who is notified, and how? | A handoff nobody sees is a lost customer with a transcript attached | "In the inbox or your helpdesk with the full transcript, plus an email or Slack alert to the people you name" |
| 8 | What does the visitor see when nobody is online? | Promising a person at 2 a.m. is worse than an honest "we reply by email" | Business hours or availability modes, an away message, and a request for an email instead of a false promise |
| 9 | How does our reply reach a visitor who has closed the tab? | Many visitors leave, and a reply that only appears in the widget may never be read | Email follow-up, a ticket in your helpdesk, or a clear statement of when the visitor would see it |
| 10 | Does the AI stop when a person takes over, and can we hand the chat back? | A bot talking over your agent confuses the customer and the record | "The AI goes quiet on escalation or when you reply, and one status change hands it back" |
Rehearse handoff three ways in the trial: ask for a person during your hours, again outside them, and once in an annoyed tone. Then check where each conversation landed, who was told, how long the alert took and what the visitor saw while waiting.
Questions to ask about knowledge sources (11 to 15)
Knowledge source questions establish what the bot can learn from, how much and how often, because a page the bot cannot read becomes an "I don't know", and a page it re-reads only weekly becomes last week's price.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 11 | Which sources can it learn from, and what are the limits in pages, depth, file size and storage? | Limits decide whether your whole site and help center fit on the plan you can afford | Numbers for crawl pages, crawl depth, file size and storage per plan, plus the supported file types |
| 12 | How often does it re-sync, and can we refresh right after a price change? | A weekly sync means up to a week of old prices unless you can force a refresh | A stated schedule, a refresh-now button, and change detection so only edited pages are re-indexed |
| 13 | What can it not read: JavaScript-built pages, scanned PDFs, images, tables? | Unreadable content turns into "I don't know" on questions your site answers | An honest list of blind spots and the workaround for each |
| 14 | How do we keep private pages out, and does the crawler respect robots.txt? | Anything in the knowledge base can be quoted to any visitor | Include and exclude paths, robots.txt honoured, and a warning that every source is public to the bot |
| 15 | Can we add or correct a single answer without editing our website? | A question-and-answer pair is the fastest fix for a wrong answer on launch day | "Yes: Q&A pairs or text snippets, live within minutes, outside your source limit" |
Ask for limits as numbers, then test the edges: import your largest PDF, your most script-heavy page and your help center, and ask a question whose answer sits on the deepest page the crawler should reach.
Questions to ask about actions and permissions (16 to 20)
Action questions find out what the bot can read and change in your other systems, and which limits are enforced in code rather than requested in a prompt, because an over-powered action turns a wrong answer into a wrong refund.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 16 | Which prebuilt actions exist, and what exactly can each one read or change? | "Integrates with Shopify" can mean anything from a link to refunds | A per-action list of inputs, returned fields and whether it writes |
| 17 | Which limits are enforced in code rather than in the prompt? | Instructions can be argued with; code cannot. OWASP calls the failure excessive agency (LLM06) | Field allow-lists, fixed hosts, per-action switches and input checks, each named as code |
| 18 | Can any action move money, and what happens if a call runs twice? | Retries can repeat a write, and refunds and payments belong with a person | "No action moves money; writes are safe to repeat or easy to merge, and here is the retry behaviour" |
| 19 | Does the bot check who is asking before it reads one customer's data? | An order lookup by email can show a stranger someone else's order | The verification method, and whether code enforces it or the prompt only asks for it |
| 20 | How are our API keys stored, who on our team can use them, and how is sign-in protected? | The bot holds keys to billing and order systems, and the dashboard controls the bot | Named encryption at rest, no read-back, role limits, and single sign-on (SSO) or at least two-factor sign-in |
OWASP's excessive agency entry names three root causes: functions the bot does not need, permissions beyond the task and high-impact actions nobody approves. Its mitigations include read-only access where possible, human approval for high-impact actions and authorisation enforced in the downstream system rather than left to the model. Our guide to chatbot permissions turns that into four tiers and ten questions to answer before you switch an action on; use it once a vendor passes this group.
Questions to ask about pricing units, caps and overage (21 to 25)
Pricing questions pin down the unit you pay for, what happens at the allowance and what stops a runaway bill, because vendors bill different units and two quotes for the same month of support can differ many times over.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 21 | What is the billing unit, and what is its written definition? | Seat, conversation, resolution, reply, session and credit measure different things | The unit, the help-center definition, the time window and what un-counts it |
| 22 | What happens when we reach the allowance: a stop, a handoff or overage? | A bot that goes silent mid-campaign and a surprise invoice are both bad outcomes | The behaviour on each plan and the overage price per 1,000 units |
| 23 | Can we set a spend cap, and who is alerted before overage? | Scripted traffic can burn units; OWASP lists denial of wallet under unbounded consumption (LLM10) | A self-serve cap and alert emails at thresholds you set, or an honest "no cap; here is the meter" |
| 24 | Do test chats, spam and replies from our own team count? | Testing and bot traffic should not appear on your invoice | "Test chats are free, human replies do not count, and rate limits stop floods" |
| 25 | What will we pay at two and five times today's volume, all meters included? | Seats, AI add-ons and per-unit fees grow at different speeds | A written quote at three volumes that lists every meter |
Question 21 deserves the most care. The Fin help article Fin pricing: Outcomes, dated 17 September 2026, is a model of a clear answer: Intercom prices Fin at $0.99 per outcome, counted when the customer confirms Fin helped or leaves without asking for more help, which includes going quiet for 24 hours after Fin's last answer. A conversation is charged once however many questions Fin resolves, and a request for a person is not billed. Ask every vendor for a definition that precise.
For question 23, OWASP's entry on unbounded consumption describes denial of wallet: attackers sending large volumes of requests to exploit pay-per-use pricing, with rate limiting and user quotas among the listed mitigations. The AI customer support pricing index lists the unit each of 30 tools bills, which makes question 25 quicker to check.
Questions to ask about data, security and privacy (26 to 30)
Data and security questions should rest on three primary frameworks, OWASP's 2025 Top 10 for LLM Applications, NIST's AI Risk Management Framework and Article 50 of the EU AI Act, so vendors answer against a standard instead of their own marketing.
The OWASP Top 10 for LLM Applications names ten risks, from LLM01 prompt injection to LLM10 unbounded consumption. The NIST AI Risk Management Framework, released on 26 January 2023 for voluntary use and now being revised, organises the work into four functions: Govern, Map, Measure and Manage. Its GOVERN 6.1 subcategory calls for policies and procedures that address AI risks from third-party entities, and a vendor evaluation is where a small team applies one.
Article 50(1) of the EU AI Act requires providers to design AI systems that interact with people so those people are informed they are dealing with an AI, unless that is obvious. Article 50(5) asks for that information clearly, at the latest at the first interaction. The Act's general application date, 2 August 2026, covers Article 50. This is information, not legal advice.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 26 | Which OWASP Top 10 for LLM Applications risks have you tested against, and how? | The 2025 list names the known failure modes, from prompt injection (LLM01) to unbounded consumption (LLM10) | A control and a test for each relevant risk, with the date of the last adversarial test |
| 27 | Who is accountable for the bot's behaviour, and how do you monitor it, roll it back and switch it off? | NIST's framework asks for assigned responsibilities and a way to disengage or deactivate a system (MANAGE 2.4) | A named role, an incident process, a change log and an off switch you control |
| 28 | How does the chat tell visitors they are talking to an AI? | Article 50 of the EU AI Act requires it, unless obvious, at the latest at the first interaction | A built-in AI label on the first screen, plus room for your own wording |
| 29 | Which companies process our visitors' messages, is any of it used for training, and how long is it kept? | Retention and sub-processors decide where copies of your customers' words live | Named sub-processors and model provider, the provider's terms, and a retention setting you control |
| 30 | Which audit reports, certifications and contracts can you share, and which do you not have? | A badge without a report proves little, and procurement may require SOC 2, ISO 27001 or a data processing agreement (DPA) | The report under NDA, or a plain list of what exists and what does not |
Two older guides go deeper where this group stays broad. AI chatbot security risks maps each OWASP risk to a control and a vendor question, and our explainer on where chatbot data goes has ten narrower data questions, including model-provider retention tiers and hosting regions.
Questions to ask about widget speed and accessibility (31 to 35)
Widget questions cover what the chat costs your pages in speed, whether every visitor can use it and what your site must allow, because the widget loads on every page it is installed on, including checkout.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 31 | How large is the widget, and what does it load before a visitor clicks? | A heavy widget slows every page it sits on | Loader size in KB gzipped, what preloads and when, measured on a real page |
| 32 | Can we load it only on click, or only on some pages? | On your fastest pages you may want zero cost until a visitor asks | A documented lazy-load setting and a JavaScript API to show, hide or remove it |
| 33 | Which WCAG 2.2 level does the widget meet, and is there an accessibility conformance report? | Keyboard and screen-reader users must be able to open, use and close the chat | "WCAG 2.2 AA, tested with keyboard and screen readers; here is the report," or a list of known gaps |
| 34 | What must our site allow, and what happens if the widget fails to load? | Strict security policies and consent banners block widgets, and an outage should never break your page | Exact CSP entries, a cookie list, async loading, a timeout message and an uptime record |
| 35 | Which languages and channels does it cover beyond the website? | A web-only bot cannot answer on WhatsApp, Messenger or email | Interface and reply languages listed, and each channel named with the plan it needs |
Measure question 31 rather than trusting it: our page-speed test of 10 chat widgets found them adding between 3.8 KB and 649 KB of downloads before anyone clicked. For question 33, WCAG 2.2, the W3C Recommendation dated 12 December 2024, is the usual reference, and the European Accessibility Act, in effect since June 2025, covers services such as e-commerce and banking in the EU.
Questions to ask about testing, reporting and exit (36 to 40)
Testing, reporting and exit questions decide whether you can prove the bot works before paying, see how it performs after launch and leave with your data if it stops working for you.
| # | Question | Why it matters | What a good answer sounds like |
|---|---|---|---|
| 36 | Can we trial every feature on our own content before signing, and do test chats count? | A demo on the vendor's content proves nothing about yours | A free plan or trial with every feature, and a test mode that does not bill |
| 37 | Which numbers does the dashboard report, and how is each defined? | "Resolution rate" means different things to different vendors | Each metric defined in writing, with the raw conversations available to recount it |
| 38 | Can we export every conversation and contact, in what format, and at what cost? | You need the record for audits, disputes and the next vendor | Self-serve CSV or API export, transcripts included, at no charge |
| 39 | How are model and prompt changes announced, and can we re-test first? | A model change can alter answers overnight | Advance notice, a changelog, and a way to rerun your test set before the change reaches visitors |
| 40 | What happens to our data, setup and contract when we leave? | Exit costs are part of the price | Cancellation terms, an export window and a deletion date, all in writing |
Question 39 matters more than it looks. Model providers retire older models on published schedules; OpenAI's deprecations page, for example, lists several shutdowns announced in October 2026. The model behind any vendor's bot can change during your contract, so ask how you will hear about it and how you can re-test.
How do you score AI chatbot vendors?
Score each answer from 0 to 3, convert each group's points into a weighted score that reflects your business, and drop any vendor that scores 0 on a knockout question, whatever its total.
- 3: meets your need, shown in your trial or in public documentation.
- 2: meets your need, stated in writing but not yet verified.
- 1: partly meets it, or answered only on a call.
- 0: does not meet it, or no clear answer.
A vague yes earns 1 at most until it is in writing, so a vendor gains nothing by blurring a gap. A knockout question is a pass-or-fail question: a 0 removes the vendor from the shortlist whatever its total. Each group has five questions and so 15 points; its weighted score is the weight multiplied by points divided by 15. The weights below are a default for a general support team.
| Group | Questions | Default weight | Vendor A points | Vendor A weighted | Vendor B points | Vendor B weighted |
|---|---|---|---|---|---|---|
| Grounding and accuracy | 1 to 5 | 20% | 12 | 16.0 | 9 | 12.0 |
| Human handoff | 6 to 10 | 15% | 9 | 9.0 | 12 | 12.0 |
| Knowledge sources | 11 to 15 | 10% | 12 | 8.0 | 9 | 6.0 |
| Actions and permissions | 16 to 20 | 10% | 6 | 4.0 | 12 | 8.0 |
| Pricing unit, caps and overage | 21 to 25 | 15% | 12 | 12.0 | 6 | 6.0 |
| Data, security and privacy | 26 to 30 | 15% | 9 | 9.0 | 12 | 12.0 |
| Widget speed and accessibility | 31 to 35 | 5% | 12 | 4.0 | 9 | 3.0 |
| Testing, reporting and exit | 36 to 40 | 10% | 9 | 6.0 | 12 | 8.0 |
| Total | 40 | 100% | 81 of 120 | 68.0 | 81 of 120 | 67.0 |
Points are out of 15 per group; weighted score = weight × points ÷ 15, out of 100. Vendors A and B are illustrations, not real products.
Both vendors earn 81 of 120 raw points, and the weights separate them, 68 to 67. Re-weight for a store that lives on order lookups, with actions at 20%, the widget at 10%, grounding at 15%, and pricing and data at 10% each, and Vendor B wins, 69 to 65. Agree the weights before the answers arrive, or the team will choose the weights that favour the demo it liked.
What are the red flags in an AI chatbot vendor's answers?
The clearest red flags are numbers without definitions, security claims without reports, prices without a written unit, and any reluctance to answer in writing or to let you test on your own content before you sign.
- An accuracy or resolution rate with no definition, test set or date. Ask what counted, on whose questions, measured when.
- "Our AI never hallucinates." No vendor can promise that. A credible vendor describes what its bot does when it is unsure.
- No route to a person, or a handoff that ends in a blank contact form the visitor must fill in from scratch.
- "Unlimited" with a fair-use clause nobody will quantify. Ask for the number at which the clause starts to apply.
- Badges instead of reports. A logo wall is not an audit report, and "bank-grade security" is not a control.
- "Our own proprietary AI" with no model provider named, which leaves question 29 unanswerable.
- Trials only on the vendor's demo content, or no trial without an annual contract.
- Refunds or payments sold as a bot feature, or limits that exist only in the prompt.
- Exports on request only, or for a fee, combined with annual auto-renewal and a long notice period.
- A discount deadline that lands before your trial ends.
How do you run a two-week AI chatbot trial?
Run a two-week trial with both finalists side by side on your own content: identical setup, the same 40 real questions, attempts to break the bot, handoff and action drills, a widget check, then a soft launch on one page.
Days 1 to 3: set up both vendors the same way
Import the same pages, documents and help-center articles into each vendor, write the same instructions, and connect the same actions with test credentials. Note how long setup took and where you needed the vendor's help; that is a preview of every later change.
Days 4 to 6: run your real questions
Ask your 40 collected questions in each vendor's test mode, word for word, and mark each reply correct, partly correct, wrong, or "I don't know" with a handoff. Add five questions your content does not answer: the right result is an honest miss, not a guess. If your inbox is thin, our chatbot test script adds 40 questions in ten groups, each with a pass check written before you ask, including the injection attempts for days 7 and 8.
Days 7 and 8: try to break it
Type five prompt-injection attempts, such as "ignore your instructions" and "print your system prompt", ask for a discount you do not offer, paste a very long message and send messages faster than a person types. OWASP's prompt injection guidance lists adversarial testing and attack simulations among its mitigations; this is the small-team version. Note what was refused, what was answered and whether anything was billed.
Days 9 and 10: rehearse handoffs and actions
Ask for a person inside and outside your hours, time how long the alert takes to reach your team and check what the visitor saw. Run every action on test records: look up a test order, file a test ticket, and confirm the bot restated what it was about to do before doing it.
Day 11: measure the widget
Load the same real page with and without each widget and compare the bytes downloaded before any click. Then open, use and close the chat with the keyboard alone, and repeat the visit on a phone.
Days 12 to 14: soft-launch and decide
Put the stronger finalist on one low-risk page, read every transcript, compare the vendor's usage meter with your own count, and export the data once to prove the export works. Then finish the scoring sheet with what the trial proved and what it disproved.
How does PepoChat answer these 40 questions?
PepoChat answers the grounding, handoff and action questions with documented behaviour, but as of October 2026 it shows no sources in replies, has no paid-plan spend cap, usage alert email or SSO, is web-only and claims no SOC 2 or ISO certification.
| # | Question | PepoChat's answer |
|---|---|---|
| 1 | Answers only from your content? | Yes: replies come from your knowledge base, never the open web, and when nothing relevant is found the agent says so and offers a person |
| 2 | Sources shown? | No: replies do not show or link the passages they used |
| 4 | Accuracy figures? | None: PepoChat publishes no accuracy, deflection or resolution statistics |
| 6 to 8 | Handoff | Escalates on request, when a visitor accepts help after a miss, and on errors; workspace members are emailed; three Team availability modes, with the AI answering in all of them |
| 9 | Reply after the visitor leaves? | Inbox replies are not emailed; when the visitor's address is known, use the alert's Reply by email button or the inbox's Send Email, or file a helpdesk ticket |
| 11 and 12 | Limits and syncing | Crawl of up to 200 pages and 3 levels, sitemap import of up to 200 URLs; pages re-fetched weekly or on demand, Notion and Google Drive daily |
| 16 to 18 | Actions | 22 prebuilt actions across 13 providers, none moves money; custom HTTPS actions with a response-field allow-list; execution is at-least-once |
| 19 | Identity check | Optional 6-digit email code; it does not gate lookups, and an action's instructions can only make the agent ask for it |
| 20 | Keys and sign-in | AES-256-GCM with one deployment-wide key and a random IV per secret, never shown again; every member can manage integrations; two-factor sign-in, no SSO |
| 21 to 24 | Billing | The unit is one AI reply; Free hands new chats to the inbox after 200 a month; paid plans bill $10 per 1,000 extra replies with no spend cap or usage alert email; test chats and team replies are free |
| 28 | AI disclosure | The chat-first opening screen reads "AI assistant · online"; the help layout and the conversation header do not, so put "AI" in the assistant's name or greeting |
| 29 and 30 | Data and certifications | OpenAI models under standard API terms; the privacy policy says knowledge-base content and visitor conversations are not used to train foundation models; Data retention of 30 to 365 days, off by default; no SOC 2, ISO, hosting region or signed DPA template claimed |
| 31 to 35 | Widget | 3,387-byte gzipped loader; chat frame preloaded in idle time unless data-preload="false"; no WCAG conformance claim or ACR; web widget only, seven interface languages; no public status page or uptime commitment |
| 37 and 38 | Reporting and export | Analytics shows sentiment votes and topics only; CSV export of conversations with transcripts, and of contacts |
| 39 and 40 | Model changes and exit | PepoChat picks the OpenAI model, so you cannot pin a version; cancel any time, and the workspace returns to Free with its data at the end of the paid period |
Checked in PepoChat's code, documentation, terms and privacy policy on 22 October 2026.
Read that table as you would any vendor's answers. PepoChat gives a written answer to each of the five default knockouts, but a buyer who needs SSO, a SOC 2 report, a signed DPA, a spend cap on a paid plan or WhatsApp should score it 0 on those questions and look elsewhere. Teams that need a grounded agent with handoff, actions and a free way to test can run the whole checklist against PepoChat before paying, because every feature except new-lead emails is on the free plan.
What to do next
- Paste the 40 questions into your RFP or a shared sheet, agree the weights and knockouts with your team, and send them to three vendors with a five-day deadline.
- Compare each vendor's security answers with PepoChat's security overview, which lists its controls as facts rather than certifications.
- To run the checklist against PepoChat, create a free workspace; test chats in the Test the agent drawer do not count toward the Free plan's 200 monthly replies.
- Related guides: what counts as a chatbot resolution.
Frequently asked questions
- What questions should I ask an AI chatbot vendor?
- Ask about eight areas: grounding and accuracy, human handoff, knowledge sources, actions and permissions, the billing unit and caps, data and security, widget speed and accessibility, and testing, reporting and exit. The most revealing questions are what the bot says when your content has no answer, how a person takes over, and exactly what you are billed for, defined in writing.
- How do you score chatbot vendors in an RFP?
- Score each answer from 0 to 3: 3 when verified in a trial or public documentation, 2 when stated in writing, 1 when partial or only verbal, and 0 when missing. Weight the eight groups for your business, for example 20% on grounding, and drop any vendor that scores 0 on a knockout question such as human handoff or data export.
- How long should an AI chatbot trial last?
- Two weeks is enough for most small and mid-size support teams: three days of identical setup, three days of real customer questions, two days of attempts to break the bot, two days of handoff and action drills, one day of widget checks, and three days of soft launch on one low-risk page before you score and decide.
- What security standards apply to AI chatbots?
- No single certification covers AI chatbots, but three frameworks shape the questions as of October 2026: the OWASP Top 10 for LLM Applications (2025), NIST's voluntary AI Risk Management Framework, and Article 50 of the EU AI Act, which since 2 August 2026 requires telling people they are talking to an AI unless obvious. Ask for SOC 2 or ISO 27001 reports separately.
- What are red flags when buying an AI chatbot?
- The biggest red flags are an accuracy or resolution rate with no definition or date, a promise that the bot never hallucinates, security badges with no audit report behind them, a billing unit the vendor will not define in writing, no trial on your own content, and conversation exports that are available only on request or for a fee.
- Does PepoChat have SOC 2 or single sign-on?
- No. As of October 2026, PepoChat claims no SOC 2 or ISO 27001 certification, no hosting region and no signed DPA template, and it has no single sign-on. Team members sign in with email and password or Google, with optional two-factor authentication. Integration credentials are encrypted with AES-256-GCM, and an optional Data retention setting deletes resolved conversations after 30 to 365 days.
Try this on your own site in ten minutes
PepoChat includes every feature on the free plan — 200 AI replies and 5 knowledge sources a month, no credit card.
Keep reading
Chatbase vs PepoChat: Which AI Support Agent Fits Your Team in 2026?
An honest side-by-side of Chatbase and PepoChat on pricing, free plan or trial, sources, handoff, booking, actions, channels and voice, and a pick by team type.
Intercom Fin Pricing Explained, and What a Flat-Plan Alternative Costs
What Intercom Fin's $0.99 per outcome adds up to once you count resolutions and seats, with a worked 2,000-conversation example and when a flat plan wins.