Skip to main content
There are three ways to create a Test Scenario: generate one from a description, build one from a production conversation, or define each section manually.

​
Generate a Test Scenario from a description

Describe what you want to test in a few sentences and the complete Test Scenario is generated for you, including mock data, customer goals, and evaluations.
  • Faster test creation: Write a simple description and get a full scenario test, with no need to build one field by field.
  • Ideal for new deployments: Builders can create tests before any real production conversations exist.
  • Lower effort to scale test coverage: Mock data, customer goals, and evaluations are auto-generated from the description, cutting manual setup work.
To generate a Test Scenario:
1

Start a new Test Scenario

Go to the Scenario Testing page and click ”+ New test scenario”.
2

Click Generate

Click Generate at the top of the scenario.
3

Describe the scenario

Add a description of what you want to test. This can be as simple as what the customer should be able to do, or as detailed as the specific evaluation criteria to apply.
4

Generate the scenario

Click Save. The complete Test Scenario is generated in a few seconds.
5

Review and save

Save the generated scenario as-is, or edit any section before saving.

​
Create a Test Scenario from a conversation

When GenerativeAgent behavior needs fixing, turn that production conversation into a test, make the change, and validate the change against the test.
  • Faster feedback loop: Spot an issue in a live conversation and turn it into a test in a few clicks, instead of writing one from scratch.
  • Higher test accuracy: Tests are grounded in real production conversations, so they reflect actual customer behavior.
  • Lower effort to scale test coverage: Mock data, customer goals, and evaluations are auto-generated from the conversation, cutting manual setup work.

​
From a full conversation

To create a Test Scenario from a conversation:
1

Find the conversation

Find a conversation where GenerativeAgent behavior needs fixing.
2

Click Create Scenario

Click Create Scenario at the top right of the conversation, then select the branch where the test should be created.
3

Review the generated test

The test is created automatically, including the description, mock data, customer goals, and evaluations pulled from the conversation.
4

Validate your changes

Make the necessary GenerativeAgent changes and validate them against the new test.

​
From a specific point in a conversation

Instead of using the whole conversation, pick any single utterance to generate a test from that exact point. Everything before that utterance is added as a conversation prefix, so the simulation picks up right where that moment happened. To create a Test Scenario from a specific point in a conversation:
1

Find the conversation

Find a conversation where GenerativeAgent behavior needs fixing.
2

Select the utterance

Pick the single utterance from which you want to generate the test.
3

Click Create Scenario

Click Create Scenario at the top right of the conversation, then select the branch where the test should be created.
4

Review the generated test

The test is created automatically, including the description, mock data, customer goals, and evaluations pulled from the conversation.
5

Validate your changes

Make the necessary GenerativeAgent changes and validate them against the new test.
Test Scenarios are scoped to the branch they are created in. See Branch Scope.

​
Create a Test Scenario manually

If you want full control over every setting, define each section of the scenario yourself. To create a Test Scenario manually:
1

Navigate to Test Scenarios

Go to the Scenario Testing page in GenerativeAgent and click ”+ New test scenario” to begin creating a new test scenario.
2

Define Basic Information

Start by providing the essential details that identify and describe your test scenario:
  • Test name: A clear, descriptive name for your scenario
  • Description: (Optional) A brief explanation of what you’re testing and what you expect to achieve
  • Tags: (Optional) Add relevant tags to categorize and easily find your scenario later
  • Test suite: (Optional) Enable it if you want to add this scenario to a test suite - a group of related test scenarios that can be run together.
3

Configure the User API Profile

The data that GenerativeAgent uses plays a critical role in how it behaves. The User API Profile defines the mock data that will be provided to GenerativeAgent when using this test scenario.You have two options to mock data: auto-generate it from a description, or define the functions yourself.
Auto-generate mock data creates function mocks using the scenario you describe and the schema of the functions in the main branch of the draft environment.To auto-generate mock data:
  1. Click Generate.
  2. Describe your scenario in a few sentences.
  3. Wait up to one minute for the profile to be created (you can close the dialog while you wait).
The system pulls mocked data schemas from the main branch of the draft environment. To test a new function, it needs to have been added to the main branch of the draft environment.
Adding details about what information the customer will have to share with GenerativeAgent and what GenerativeAgent will have to do will significantly improve the ability to auto-mock the appropriate APIs.
If you know the specific data you want to mock for a scenario, you can manually define the functions and mock data.
  1. Click + Select functions.
  2. Choose the API calls you want to mock (e.g. getAccountInfo, confirmCode).
  3. For each function, provide the mocked request and response in JSON.
Add variants to mocked dataTo simulate different API responses under the same function:
  1. Click + Add variants at the bottom of a function.
  2. Define alternative request parameters and corresponding response schema.
  3. Save to include the variant in your scenario.
You may need to update the mock data after creating a Test Scenario. After creating a Test Scenario, ensure your mock data aligns with your testing needs.This can be done by updating your scenario when auto-generating the mock data or manually adding the functions and mock data.
Often APIs have an implicit or explicit date and time when the system performs them. By default, GenerativeAgent assumes the customer interaction is happening at the date and time the scenario is run and updates timestamps in the API profile accordingly.You can override the assumed timestamp of the interaction.
4

(Optional) Set Input Context

The Input Context defines the initial data and configuration that your system provides to GenerativeAgent when a conversation begins. This ensures your test scenarios accurately simulate real-world interactions.The input context consists of two main components:
  1. Input Variables: JSON data that provides context about the customer, conversation, or session
  2. Starting Task: The specific task or flow that GenerativeAgent should begin with
Ensure your input context matches exactly how your production system passes data to GenerativeAgent. This guarantees your test scenarios will match real conversations identically.
5

(Optional) Define Simulation Details

If you are using the automatic mode, configure the customer’s behavior and goals in the simulation details. This is especially useful for testing how GenerativeAgent handles different customer personalities, levels of information, and goals.
  • Start of conversation: Define who initiates the conversation: the customer or GenerativeAgent.
  • Customer Goals: Define the motivations of the simulated customer. The customer will attempt each goal in sequence; if it fails, it may escalate or skip.
  • Information the customer knows: Define the identity or account data the customer can provide. To test how GenerativeAgent will handle a customer who does not have their account number or a critical piece of information, just leave it off the list.
  • Customer personality: Define the tone and style for your simulated customer (optional). eg. rude, insistent, confused, etc. Click Revert to default personality to restore.
  • Max number of turns: Set the maximum number of conversation turns the simulation can run. By default, it is set to 15 turns, but you can adjust it based on your testing needs. This is especially useful for testing longer conversations or ensuring that the simulation runs until a specific outcome is reached.
6

(Optional) Add Evaluations

Toggle evaluations on to run evaluation logic after the conversation ends.
Define preconditions or milestones that must be met in the conversation for the evaluation to run. This is useful to disqualify cases where the conversation didn’t reach the part you’re testing.If left blank, the evaluation runs by default.Example:
“Customer asked for their bill amount”
Define one or more evaluation checks to apply to the conversation.Examples:
“The Agent provided the current month’s bill amount of $144.72” “The Agent did NOT offer a discount”
Evaluation results appear in the Previewer, side-by-side with the conversation after clicking on the Results button. If you update your evaluation or applicability criteria later, you can return to the same conversation in the Previewer and click Run Eval Again to re-run against the updated test.
7

Save your Test Scenario

After completing these steps, save your Test Scenario and start running it in either manual or automatic simulation mode.
You can now preview the test scenario directly from the test file after saving your changes.

​
Next steps

Run Test Scenarios

Learn how to run your Test Scenarios in the Scenario Runner and review the results.

Test Scenarios

Learn what a Test Scenario defines, how branch scope works, and how to export and import scenarios.