Agent analytics

Agent analytics: what it is, platforms, courses, and benefits

Agent analytics is the practice of measuring, monitoring and analyzing AI agents in production, from observing failures and correlating them with root causes to suggesting fixes and continuously evaluating changes, so teams can control cost, reliability, auditability and governance.

Berca Akbayır

Member of Technical Staff13 min read

On this page
  1. What is Agent Analytics? What are the fundamentals, core concepts?
  2. Benefits of Agent Analytics
  3. How do we do it?
  4. Leading the Agent Analytics Platforms: Salesforce and Pendo Examples
  5. Agent Analytics Courses & Certifications
  6. FAQs
  7. References

TL;DR

  • Agent analytics is the practice of measuring, monitoring and analysing the behavior of agents in production.
  • The key to agent analytics is being able to see what is going wrong through an observability layer, correlating causes with findings, producing improvement suggestions, and continuously evaluating the agent.
  • Agent analytics is essential for control, reliability, auditability and governance.
  • Agent analytics is a foundational layer that provides full visibility into an agentic system, enabling the collection of insights regarding data trends, primary cost drivers, recurring failure patterns, and key indicators for system optimization.

What is Agent Analytics? What are the fundamentals, core concepts?

Agent Analytics is the practice of measuring, diagnosing and improving the behavior of AI agents in production, so teams can control their cost, reliability and quality. Agents reason over multiple steps and act through tools, so their uncertainty compounds with every step, and a wrong action can cause real damage, such as an incorrect refund or a corrupted record. Agents can be constrained but never made fully deterministic, so the practical question is how to manage the uncertainty that remains. This post breaks agent analytics into layers, from observability to automated resolution, then covers the platforms implementing them and how to learn the field.

Layer 1: Observability

To manage remaining uncertainty, one must first measure and analyze it. This allows for identifying what went wrong and why, while providing full visibility into the "black box" operations of AI agents. Key metrics to track include cost, success rate, system latency, and conversation quality. These constitute the observability layer, the foundational tier of of AI agent analytics. This layer addresses questions such as: What is the cost per interaction? What is the latency of the responses? How accurately does the agent interpret user requests, and how successfully does it manage them? These metrics are sufficient to gauge the overall health of an agentic system and serve as a starting point for asking the right questions.

Layer 2: Correlation

The second layer is the correlation layer. This layer emerges as one begins to ask the right questions based on the metrics evaluated in the first layer; it involves investigating the root causes of the issues identified. For instance: Under what circumstances does the agent incur higher costs? Does it enter a loop when handling specific types of requests? Are there particular user requests that trigger a higher error rate? Are the observed latencies inherent to the process, a result of the surrounding system, or due to optimization issues within the agentic system itself? These are the types of questions that arise and require resolution following any temporal, periodic, or isolated fluctuations in high level metrics like cost, success rate, and quality.

Layer 3: Solution

The third layer is where solutions are generated. Once the root causes of issues have been established through correlation analysis in the second layer, the agent analytics platform should identify and propose proactive solutions to address these problems. Once the bottlenecks of an agentic system have been identified, a solution is determined for each specific point. This approach minimizes the number of issues that need to be addressed. These bottlenecks may stem from the agent's design, the system in which the agent operates, or the user.

Continuous Evaluation

The introduction of a third layer (proactive solution generation, or automated problem solving) gives rise to a new requirement, solution tracking and versioning. Agents updated with proactive solutions generated through agent analytics must be versioned, the implemented solution must be tracked, and its impact must be validated within the production environment. This process is known as continuous evaluation and backtesting. Before a generated solution is deployed to production, it is tested against historical data to verify its effectiveness; once in production, its performance is monitored to ensure that the quality is maintained. If a new, versioned solution fails during backtesting or in the production environment, the system reverts to the previous version of the agent.

The three layers of agent analytics as a pyramid: observability, correlation and solution, with continuous evaluation looping back to observability

Evaluation of Needs in Agent Analytics

The need for AI agent observability at an enterprise level is not a new concept. In the late 2010s, companies striving to derive value from data by deploying machine learning models into production addressed the inherent uncertainties of these environments through MLOps. As AI models were developed, teams monitored whether models with various configurations provided accurate responses to data, tracked shifts in underlying data assumptions, and measured performance in production. Furthermore, they monitored costs and latency, established triggers, and retrained or updated models as necessary, practices that remain in use today. However, with the evolution of agents, Large Language Models (LLMs) have moved beyond simple Q&A tasks; they now utilize tools, manage workflows, and handle requests in production environments. In response to these developments, the framework known as LLM Analytics has matured into "AgentOps". When only LLMs were in the production, high level cost analysis and Q&A quality evaluation were enough in LLM Analytics framework as LLM observability. The focus has shifted from monitoring only cost and basic quality metrics to tracking "agent behavior," and from monitoring data drift in MLOps to analyzing user intents. As agents take on greater responsibilities bringing with them increased uncertainty and new types of costs (such as the functional costs incurred when an AI produces an incorrect output, in addition to the direct operational costs of the AI itself) the conversation has expanded beyond mere observability and versioning to include the concept of automated resolution.

Benefits of Agent Analytics

The main benefit of agent analytics is turning the uncertainty of agents into something that can be measured, explained and improved. Each layer of the framework brings its own value:

Cost control. Agents do not only consume tokens. They consume them in loops, retries and unnecessary tool calls. Agent analytics shows which requests, intents or paths are responsible for the cost, and whether that cost results in a successful interaction or not. In other words, the question changes from “how much does the agent cost?” to “how much does a successful interaction cost?”

Reliability. Loops, tool failures and error prone request types are detected before they turn into user complaints. Instead of reacting to individual failures, teams see the patterns behind them.

Faster and safer iteration. Every change to the agent, whether a prompt, a tool description or a model update, is versioned, backtested and monitored. Decisions are made based on data instead of intuition, and a failed change can be rolled back before it causes damage.

Governance and auditability. Traces show what the agent did, which tools it used and why. This is essential in regulated domains, where an autonomous system’s actions must be explainable.

Intent Management. The distribution of user intents shows what users actually expect from the agent, which requests it cannot handle, and where the product should evolve next.

The five benefits of agent analytics: cost control, reliability, faster and safer iteration, governance and audit, intent management

How do we do it?

Easy Integration

Ratel works from the telemetry an agent already produces through any OpenTelemetry exporter, so nothing in the agent changes. Each model call's input tokens are split into what the agent spent context on: skills, tools, history, memory and user input. Agents that route tool retrieval through the Ratel SDK add what plain telemetry cannot show: each search, its ranked results, and which tool the agent then chose.

Observability & Analytics Layer

The “Overview” dashboard shows cost per day and per tool, tokens by source, latency and error rates, and how much of each run was reclaimable. Traces opens any run step by step.

Three analyses turn those metrics into answers. “Intents” groups what users asked for and attributes cost and outcome to each group, so a rise in a top level metric can be traced to one kind of request. “Agent health” runs around thirty detectors over the turns: searched again for the same thing, searched and ran nothing, repeated the same call, resent a failed call unchanged, looped on errors, confused two look-alike tools. Alongside these it reads the agent's own self corrections, such as retrying, backtracking or giving up, and runs sentiment analysis on the users' reaction and agent behavior in each turn. Comparing negative sentiment in turns where the agent corrected itself against clean turns shows whether a detected problem actually reached the user, which is a far better read of success than a completion flag. “Match quality“ checks, for every search, whether the right tool was in the list and how far down it sat.

Improvement Suggestions

“Fixes” is one ranked list drawn from usage patterns, retrieval failures and health issues that name a concrete change, plus "write a skill" where a frequent intent has none. The kinds are specific: rewrite a description, tighten a schema, tell two siblings apart, merge, retire, fail clearly, preload a prerequisite. Fixes with a measured effect on solid evidence rank first. A description rewrite can be applied from Ratel as a reversible override the runtime picks up on its next fetch. Every other fix is handed over as the exact change to make. Beyond one off fixes, “Adaptive ranking” is a standing improvement: Ratel learns from production which tools each kind of ask actually leads to and builds a graph the runtime can rank from, so the right tool rises to the top of search results without anyone editing the catalog. The graph is first watched in shadow against the current ranking and goes live only when it measurably ranks the tool the agent used higher.

Continuous Evaluation

A rewrite is backtested by replaying historical searches through the same retrieval engine the runtime uses, so its effect on match quality is known before deployment. Once the team ships a new version, “Calibration” reports the split or shadow test between the old and new arms: agreement between rankings, and the gaps in error rate, latency, cost, sentiment and intent outcome, with flags for imbalance or too few samples. Promotion and rollback stay the team's decision, made on those numbers, and the lineage from trace to fix to shipment is kept so the effect can be measured afterwards.

Ratel's loop: your agent's OpenTelemetry traces and Ratel SDK searches feed Observe, Diagnose, Fix and Evaluate, which loop back to the agent

An End to End Example

Consider a task management platform that exposes its product to coding agents through an MCP server with close to two hundred operations. On the Ratel dashboard, input token cost per session had risen 47% in three weeks while the task completion rate slipped from 91% to 84%. Segmenting sessions by intent showed the increase came almost entirely from planning questions such as "what should I work on next", and Agent health flagged the same sessions for repeated catalog searches, failed invocations and users rephrasing the same request. The traces revealed the cause: for one planning question the agent called four overlapping list tools in a row, because their descriptions did not say which one to use when several filters apply, and each extra call pulled thousands of tokens of task rows into context. The Fixes view proposed the change with the description diff already drafted: name one tool as the single entry point for combined filters and state the one case each of the others covers. The team saved it as a new catalog version and ran it through Calibration on 20% of real traffic against the previous version. Tool calls per planning question fell from 4.3 to 1.1, input tokens per session dropped 38%, cost returned below its original level and completion recovered to 93%, so the new version was promoted to all users. Had quality dropped, the arm would have been rolled back without any customer-visible change. One fluctuation in a top-level metric led to an intent, a trace, a one line description fix and a validated rollout, which is the full cycle of agent analytics.

Leading the Agent Analytics Platforms: Salesforce and Pendo Examples

As the framework of observability, correlation, and automated resolution matures into production systems, teams require dedicated infrastructure to manage agent telemetry, reasoning traces, and user intent. Salesforce and Pendo are two examples of how this architecture is applied in practice. They were chosen because they cover two different sides of it: backend enterprise logic and frontend user experience.

Salesforce and Agentforce Analytics Integration

To operationalize agentic systems at an enterprise scale, tracking must tie directly into core business workflows. Salesforce exemplifies this by moving beyond passive chatbots into autonomous, action taking digital labor embedded natively into enterprise CRM environments via Agentforce. Agentforce Analytics maps onto our core framework by connecting agent execution data directly to enterprise context. It fulfills the observability layer by automatically capturing user interactions, multi step reasoning paths, and task completion rates using Salesforce Data Cloud objects when enabled. It supports the correlation layer as well: because agents operate directly on Salesforce objects, flows, and APIs, administrators can trace error propagation, data consumption, and operational usage back to specific workflows natively within Salesforce Setup. This creates a feedback loop where performance telemetry informs agent adjustments, ensuring that autonomous customer service and sales workflows align with enterprise governance.

Pendo Agent Analytics

While enterprise CRM platforms govern backend operations, product led organizations require visibility into how end users actually engage with AI features inside web and mobile applications. Pendo Agent Analytics is a strong example of tracking user facing agent interactions, connecting traditional product behavioral telemetry with conversational AI tracking. Rather than evaluating an agent in a vacuum, Pendo maps individual prompts, token costs, and tool calling traces directly to downstream user behavior, fulfilling the observability layer through core features like product behavioral context, session tracking, and full conversation transcripts. Furthermore, Pendo addresses the correlation layer by automatically surfacing friction points, such as unsupported requests, system errors, and rage prompts from frustrated user inputs. Its built in experiments feature allows product teams to run A/B tests evaluating different prompts or agent versions against real user outcomes, enabling continuous evaluation and version control directly in production.

Agent Analytics Courses & Certifications

Agent analytics is still quite a new concept, so its best practices and standards are not settled yet; they are being defined by the companies and organizations that build and operate agentic systems at scale. Therefore, the most direct way to build expertise in agentic systems and agent analytics, and to develop deep know how, is to follow the courses and earn the certifications of these pioneers.

Here are some of the most recognized courses and certifications that cover agent analytics while providing the fundamentals of agentic AI.

AWS Certified Generative AI Developer – Professional (AIP C01) by AWS

AWS certified GenAI Developer is a certification program that covers advanced technical expertise in building and deploying production ready AI solutions using AWS Services like Bedrock. This certification provides a reliable way to identify and verify developers who can move beyond PoC to build production grade generative AI solutions that deliver tangible business results while maintaining security and cost efficiency.

certified-generative-ai-developer-professional

Building Agentic Systems on AWS (Developer Series) by AWS

This course series that prepared by AWS provides advanced learning program to learn the fundamentals of AI agents, multi agent architectures and orchestration, agent evaluation and observability for production level agentic systems. It is great course for who asks ‘how to build an enterprise level agentic system, how to evaluate and observe the performance, how to keep it safe‘.

Build AI agents with AWS | AI agent learning series | AWS Marketplace

Google Cloud AI Agents: Foundations to Enterprise Scale by Google Cloud/Coursera

This course provides comprehensive course about building, deploying and scaling the agentic systems for beginners. Participants can learn the fundamentals of agentic AI and essentials of developing the agentic system at enterprise level. At the end of this course, participants can earn an employer recognized certificate from Google Cloud

Google Cloud AI Agents: From Foundations to Enterprise Scale

LangChain Certified Agent Engineer (LCAE) by Langchain

LangChain provides an end to end training course and certification in building, testing, deploying and monitoring the agents. It is one of the best certifications that proves the ability of managing all agentic system lifecycle.

Become a LangChain Certified Agent Engineer

FAQs

  • Is ChatGPT classified as an AI agent or an LLM?

    ChatGPT is an application built on top of the OpenAI’s GPT models to handle the Q&A tasks with users. Meanwhile, it can act as AI agent by using tools like web search, code execution to handle the request during the conversation.

  • What are the primary types of analytics used by agent analytics?

    The primary analytics can be grouped as following structure:

    • Descriptive (what happened?): cost, latency, success rate and conversation quality, i.e. the observability layer.
    • Diagnostic (why did it happen?): root cause analysis by intent, loops and error patterns, i.e. the correlation layer.
    • Prescriptive (what should be done?): proposing and validating fixes, i.e. the solution generation and versioning layers.

    Additionally Predictive (what will happen?) can be added to anticipate cost spikes, failure prone requests or quality drops before they occur from the pattern of historical traces.

  • What is the main difference between an AI agent and an analytics agent?

    Analytics agent is type of AI agent that specialized in data analytics, reporting and producing insights for specific domain. In other words, every analytics agent is an AI agent, but not every AI agent is an analytics agent.

  • What specific user interactions does Pendo Agent Analytics track?

    Pendo Agent Analytics tracks the prompts users send to an AI agent and, with full conversation capture, the agent’s responses, along with the tools, models and tokens used. From these, it measures adoption, quality signals such as issues, rage prompts, unsupported requests and errors, and user feedback, and connects them to what users do elsewhere in the product.

References