Agent tests

An agent test is a saved conversation that checks the agent still behaves after you change instructions, knowledge or actions. You write the visitor's messages and what success looks like, and optionally the actions the agent must call. Each run sends the messages to the real agent in a free, hidden conversation, and a judge model grades the transcript as passed or failed, with a reason.

Create a test

1. Open Agent tests and click New test

   Agent tests is under Configure in the sidebar. Give the test a Name of up to 60 characters.

2. Write the visitor's messages

   Visitor messages (one per line, sent in order): up to 8 lines, each up to 1,000 characters. The lines are sent as written, whatever the agent replies, so include the details it will ask for, such as an order number.

3. Say what success looks like

   What does success look like?, up to 1,000 characters. Be concrete: "The agent asks for the order number and total before saving the refund request, confirms it was submitted, and never promises a refund amount."

4. Pick the actions the agent must call (optional)

   Under Actions the agent must call, tick up to 10. The list holds Escalate to a person, Collect contact details, each custom action by name, each data form as Collect data: Refund request, and each procedure as Procedure: Cancel a subscription.

5. Click Create test, then Run

   Run on the card starts one run; Run all starts every test that is not already running.

How a run works

1. A new test conversation opens with a visitor named "Test visitor" and your widget's greeting.
2. The first line is sent through the same path the widget uses, so the agent answers with your live instructions, knowledge, custom actions, data forms and published procedures. Drafts and switched-off actions or forms are not used.
3. The run waits for the agent's reply and for its turn to finish (tool calls done, final text written), then sends the next line.
4. If the agent hands over to a human, or the conversation is resolved, the script stops there.
5. A judge model reads your success criterion, the transcript (tool calls left out), the actions the agent called and the ones you required. It is told to judge only the agent and to be strict: invented facts, ignored instructions, the wrong tone or an unanswered question fail. It returns pass or fail, a score from 0 to 100 and a reason of at most two sentences.

The pass rule

A run passes only when all three hold:

• the judge says it passed;
• the score is 70 or more;
• every action under Must call was called at least once.

A missing action fails the run whatever the judge thought: the score is capped at 69 and the reason lists the actions not called. Names are compared ignoring capitals and extra spaces. An action counts as called even when the call returned an error.

What runs for real

A run is a test chat, like the Test the agent drawer:

• The agent's replies are free and do not count toward your AI replies.
• The conversation is hidden from the inbox unless you switch on Include test chats, and it is left out of Analytics, including the Action calls card.
• Data forms and contact details the agent collects are not saved: nothing lands under Collected data or Contacts. The webhooks you set up on the Developers page are not called for test conversations.
• Custom actions, including the ones integrations added, send their requests for real, and booking tools book real meetings. Use test-mode keys or a test customer for anything that writes.
• A hand-over is real too: the test conversation is marked escalated and your team gets the usual hand-over email.

Results

The page header shows Tests, Passing, Failing (failed runs and errors) and Pass rate, all based on each test's latest run. Each card shows that run's result:

Runs on the card lists the last 20 runs with the result, the reason or error, the start time and the duration. Click a run to see Actions called and the transcript. Deleting a test deletes its runs.

Time limits

• The agent has 75 seconds to answer each line, or the run ends as Error with The agent did not answer "..." within 75 s.
• After the reply, the run waits up to 30 seconds for the turn to finish, then goes on with what it has.
• A run still unfinished 13 minutes after it started is closed as Error: "The run stopped before it finished, so there is no verdict. Run the test again."

Limits

• 50 tests per workspace.
• 8 visitor messages per test, each up to 1,000 characters; success criterion up to 1,000; up to 10 required actions.
• 20 runs kept per test.
• Run all handles up to 20 tests at a time.
• Runs are limited per workspace to 60 an hour. Run uses one, and Run all uses one for every test it starts.
• Agent tests are available on every plan.

Common errors