Testing Your Agents¶
Test Scenarios lets you test an agent with a simulated caller before real customers reach it. The simulated caller phones your agent, holds a whole conversation, and Vyom then checks how the agent handled it: whether it ended the call properly, handed over to a person when it should, or used the right tool.
Where to find it: Console menu, Test Scenarios.
Who can use it: Tenant Developer and above. Other roles don't see the page.
Test runs use credits
A test run is a real conversation with your agent's real AI models, so it is charged like a call. A run of a few minutes typically costs a few credits.
How a test works¶
- You pick an agent and a scenario. A scenario describes the caller: who they are, why they're calling, and how hard they are to serve.
- The agent runs exactly as it does for real callers, with its own instructions, voice, tools and knowledge bases. A scenario never changes the agent.
- Test calls are kept apart from real ones. They don't appear in Agent Call History unless you tick Include test scenario calls, and they're left out of Agent Insights.
Run a test¶
- Under Agent under test, select the agent.
- In the Scenario library, select a scenario. Use the tags at the top to narrow the list, for example identity or handover.
- Check the Details this caller can give panel (see below). If it says NEEDED TO RUN, fill it in and save it first.
- Select Run Scenario.
The conversation appears as it happens, turn by turn. While it runs you can:
- Listen to hear the call live. You only listen; neither side hears you.
- Take over as human to join the call as a person, the way a supervisor would after an escalation.
- Stop to end the run early.
A run ends when the call ends or when it reaches the scenario's time cap.
Read the result¶
When the run is scored, a verdict appears above the transcript: PASS or FAIL, with a count of the checks that passed, failed, or were not exercised. A check is not exercised when the call never gave it a chance to happen, for example a check on what happens after a person joins, when no one joined.
Each check is listed with a one-line explanation, for example:
| Check | What it looks at |
|---|---|
| call_end | The call ended on its own, before the time cap. |
| escalated | The agent handed the call to a person when the scenario expected it to. |
| agent_silent_after_handover | After handing over, the agent said its handover line and then stayed quiet. |
| human_linked | When a person joined, the caller was connected to them. |
| outcome | The scenario's overall goal, built from the checks above. |
Some checks are marked judge. They are an AI reviewer's opinion of the conversation. They never change the PASS or FAIL verdict, which comes only from the checks above. A judge's doubt about a run that otherwise passed shows as CONCERN.
Below the checks, the tools the agent used during the call are listed in order. This is useful when a check surprises you.
Past runs¶
Past runs lists earlier runs of the selected scenario, newest first. Switch between This agent and All agents to compare how different agents handled the same caller. Select a run to open its transcript and result again.
Run many at once¶
The Batch runs tab runs several scenarios against several agents in one go, and reads the whole set as one result. This is how you check that a change to an agent's instructions improved things without breaking something else.
- Open the Batch runs tab.
- Tick the agents and the scenarios you want. Every agent is run against every scenario, so three agents and four scenarios is twelve calls.
- Optionally give the batch a label (for example
nightly-smoke) so you can recognise it later. - Repeats runs every combination more than once. Leave it at 1 unless you are chasing an inconsistent result — see the note below.
- Concurrency is how many calls run at the same time. Leave it at 1 unless you know your agents have capacity to spare; each test call occupies a slot alongside your real calls.
- Select Run batch.
A batch multiplies the cost
Every member of a batch is a real conversation charged like a call. The summary line above the button tells you how many calls you are about to make before you start.
Before anything runs, Vyom checks the whole selection and tells you what it found:
- A pair that cannot run blocks the batch, and the batch is all-or-nothing. This is deliberate: a suite quietly missing three of its forty members is not the same suite you compared against last week. Fix what it names, or untick that pair.
- A warning still runs, but tells you something in it will have less to say than you might expect.
While the batch runs you can watch its progress and Abort it. Aborting stops members that have not started yet; ones already in progress are stopped too.
Read a batch result¶
The batch gets its own PASS or FAIL, and it is strict: a batch passes only if every member that counted passed.
| What you see | What it means |
|---|---|
| Passed 9/12 (75%) | Nine of the twelve calls that counted passed. The count is always shown with the total, because a percentage on its own hides how few calls it came from. |
| Not counted | Calls left out of the score on purpose. A call the test harness or a provider broke is not evidence about your agent, and a call whose setup never happened didn't test what it claims to. Leaving them in would make an outage look like your agent getting worse. |
| Never exercised | A check that had nothing to score in any call of the batch. This is untested, not passing — it usually means no scenario in your selection reaches that part of the conversation. |
| Judge | The AI reviewer's count, shown next to the verdict. As with a single run, it never changes PASS or FAIL. |
The results matrix has one row per call and one column per check. Each column header shows how often that check passed and out of how many. Select any row to open that call's full report and transcript, exactly as you would a single run.
At the bottom of the summary, each agent's instructions are identified by a short code. Two batches showing the same code ran the same instructions, which is what lets you say a result changed because you edited a prompt rather than just that it changed.
Repeats, and knowing what a change is worth
Some results are not perfectly repeatable: the AI judge occasionally changes its mind on a borderline call, and a difficult caller does not always behave difficultly. Running the same selection twice with no changes at all tells you how much a result moves on its own. A later difference smaller than that tells you nothing.
The scenario library¶
Vyom comes with built-in scenarios for common situations, including:
| Scenario | What it tests |
|---|---|
| Happy path | A cooperative caller with a routine request. A baseline to compare against. |
| No caller ID | The agent has to ask who is calling. |
| Caller volunteers nothing | The caller only gives their details when asked. |
| Caller gets their own details wrong | A detail is wrong the first time. The agent should ask again, not guess. |
| Reference number comes through garbled | The agent should ask the caller to repeat or spell it. |
| Caller asks for confirmation twice | The agent should confirm again calmly. |
| Caller asks for a human | The agent should hand over to a person. |
| Question it has no answer for | The agent should say it doesn't know, not make something up. |
Select a scenario to see its details: The caller (name, calling from, situation), How hard they are to serve, and Run settings (time cap, language, whether a person joins, speech style).
Details this caller can give¶
Many scenarios need the caller to identify themselves, for example with a booking reference or an account number. Those details depend on your systems, not on the scenario, so you set them per agent: the same scenario can then run against every agent you have.
- Select the agent and the scenario.
- In Details this caller can give \<agent>, enter each detail's name and value, for example
booking_referenceand a real test booking. Vyom suggests detail names from the agent's own tools. - Select Save for \<agent>.
The panel shows CONFIGURED once saved, NEEDED TO RUN if the scenario can't run without them, or OPTIONAL if the scenario never asks the caller to identify themselves.
Make your own scenarios¶
Built-in scenarios are never changed. To adapt one:
- Select it and choose Clone & edit (or New From Selected at the top).
- Change the Scenario name, the Caller's name, What this tests, The caller's situation, Calling from (leave it blank for no caller ID), the Time cap, and How hard this caller is to serve.
- Save. Your copy appears in the library.
You can delete your own scenarios, but not the built-in ones.
Export saves a scenario as a file. Import (top of the page) loads one or more scenario files, for example to move scenarios from one Vyom account to another. If a file can't be imported, the message says which part of it is wrong.
If something goes wrong¶
- Run Scenario is greyed out. Select both an agent and a scenario. If the details panel says NEEDED TO RUN, save the caller's details first.
- "A run is already in progress." Wait for the current run to finish, or select Stop.
- The run fails on call_end. The call reached the time cap. Check that the agent hangs up after saying goodbye (see Agent Actions), or choose a longer time cap in your own copy of the scenario.