Conversations

As you add use cases and refine GenerativeAgent’s configuration, you can review, analyze, and fine-tune how the GenerativeAgent handles real customer interactions. The Conversations interface enables you to:

• Find specific conversations using search and filter options
• Share conversations with others
• Review how GenerativeAgent handled customer interactions by going through the conversation transcript or playing back the conversation.
• Use Evaluators to review and analyze conversations.
• Analyze GenerativeAgent’s decision-making process via model actions

​

Access Conversations

Once you have access to Conversations, follow these steps to navigate to the interface:

1. Sign in to the AI-Console.
2. From the AI-Console landing page, navigate to the “GenerativeAgent” section in the left-hand menu.
3. On the left-hand panel, navigate to Conversations.

​

Find conversations

In the Conversations interface, you can find specific conversations using the search and filter options. You can either combine multiple search terms and filters to narrow down your results or use them individually to explore different aspects of your GenerativeAgent interactions. When you open a conversation, you can see the follow:

• Full transcript of the interaction between the customer and GenerativeAgent.
• Model actions that show how GenerativeAgent processed the conversation.
• Flags that indicate potential quality issues in the conversation under Quality.
• Goals and their completion status under the About tab.
• A button to copy the conversation link for sharing.
• Summary of the conversation under the Summary tab, which provides a brief overview of the interaction.
• Structured data that was extracted from the conversation, which can be used for further analysis and insights.

​

Search conversations

Use the search bar to find conversations containing specific words or phrases. Enclose terms in quotes for exact matches.

​

Filter conversations

You can filter conversations using the following criteria:

• Agent: Filter by the GenerativeAgent handling the conversation
• Agent ID: Search for conversations by a specific GenerativeAgent ID
• Conversation ID: Search for a specific conversation
• Customer ID: Find conversations involving a specific customer
• Date range: Select specific time periods
• Escalation from GenerativeAgent: Find conversations that were escalated to a human agent
• Functions: Locate conversations that called particular APIs
• Flags: Find conversations flagged for quality issues
• Goal Completion: Filter conversations by whether customer goals were met
• HILA Consult: Find conversations that involved Human in the Loop assistance
• Intent: Search for conversations involving specific intents
• Internal Conversation ID: Search using internal conversation identifiers
• Structured Data: Filter conversations based on extracted structured data points
• System Transfers: Find conversations that involved system transfers
• Task: Find conversations where specific tasks were performed

​

Finding Flagged conversations

Find both manually flagged conversations and those flagged by automated systems by using the “GenerativeAgent Flags” filter from the filter options. You can further refine your search by selecting specific flag types (manual or automated), tags, and severity levels.

​

Review conversations

You can review the Conversation by playbacking it to understand the flow of the interaction and how GenerativeAgent responded to different customer inputs. Also, you can Manually annotate specific turns or messages that require attention to provide feedback, identify issues, or highlight important moments in the conversation.

​

Playback a conversation

You can play back a conversation to listen to how GenerativeAgent interacted with the customer over time. Conversation Playback is useful for understanding the flow of the conversation and how GenerativeAgent responded to different customer inputs. The Conversation Playback enables you to:

• Play, pause, and navigate through the conversation timeline.
• Skip forward or backward by 15 seconds.
• Play the conversation at different speeds such as 1x, 1.5x, and 2x.

​

Manually annotate a conversation

You can add manual annotations to a conversation to flag specific turns or messages that require attention. This is useful for providing feedback, identifying issues, or highlighting important moments in the conversation. The Manual Annotations can be done by using the Manual Annotation Evaluator which allows you to add flags and comments to specific turns in the conversation. Manual annotations include:

• Flags: Mark specific turns or messages with predefined categories and severity levels (e.g., critical, major) to indicate quality issues or areas for improvement.
• Comments: Add detailed comments to provide context and rationale for the flags, which can help in understanding the issue and guiding improvements.
• Flag refinement: For turns already flagged by automated systems, you can confirm, recategorize, dismiss, or update the rationale to refine the annotation and improve evaluator training data.
• Classification: Use tags to classify the type of issue (e.g., policy violation, incorrect information, escalation required) for better organization and analysis.

You can find the manually flagged conversations by using the appropriate filters. For more information on manual annotations, see Manual Annotation Evaluator.

​

Analyze conversations

Use Goal Completion and Conversation Monitoring evaluators to analyze conversations based on specific quality metrics. Evaluators help you analyze the quality of conversations, goal completion, and other important aspects of GenerativeAgent’s performance.

​

Goal Completion

Evaluate whether customer goals were successfully achieved during conversations. Goals are specific objectives that customers aim to accomplish during their interactions with GenerativeAgent, such as resolving an issue, obtaining information, or completing a transaction. You can evaluate goal completion by: See Goal Completion for detailed instructions on how to use this evaluator.

​

Conversation Monitoring

The Conversation Monitoring evaluator enables you to monitor conversations and identify issues that may impact the quality of customer interactions. There are quality evaluators that assess each turn for adherence to configured tasks and knowledge. When evaluators detect issues, they:

• Flag the conversation
• Highlight problematic utterances
• Provide rationale for flagging

You can find the system flagged conversations by using the appropriate filters. Once a conversation is flagged, you will find an inline indicator near the particular flow. You can analyze them in the Quality tab of the conversation review interface. The flags can be customized by changing their severity level (e.g., from “major” to “critical”) or dismissing them if they are false positives. See Conversation Monitoring for detailed instructions on how to use this evaluator.

​

Analyze Model Actions

Once you have found a conversation, you can see exactly how GenerativeAgent makes decisions by viewing its internal reasoning process. Model actions are the input, knowledge, api calls, reasoning, and output of GenerativeAgent’s model while handling the customer interaction. The information in the model actions can drive how you update the configuration of your tasks and functions. When looking at a conversation, model actions are displayed inline with the conversation flow. This allows you to understand exactly when and why the AI made specific decisions during the interaction. To analyze model actions:

1. Open a conversation
2. In the center panel, enable the model actions you want to review
   
3. View the AI’s reasoning process inline with the conversation, showing the chronological flow of decisions
4. Click any model action to see detailed information. This example shows a function response.
   
   You can also see the “Raw” JSON interaction between GenerativeAgent and the function.

​

Model actions categories

Model actions are categorized into the following:

​

Share a conversation

You can share a conversation with others by clicking the “Copy Link” button when viewing a conversation. You can also share your current filtered view by copying the URL of your current page.

​

Next Steps: Explore Evaluators

Evaluators help you review and analyze conversations based on specific quality metrics. You can use evaluators to have an in-depth analysis of GenerativeAgent’s performance.