What is chatbot analytics?
Chatbot analytics is the analysis of the conversations between users and chatbots. It covers the collection, analysis and interpretation of the data generated by these conversations. This data offers valuable insights into the quality of interactions, customer needs, areas for service improvement and business opportunities.
Key metrics for chatbot analytics
The key metrics tracked in chatbot analytics can be grouped into three sections. First, user engagement should be measured to find out whether the chatbot is being used and whether users come back after their first visit. Second, goal completion should be tracked. Can the chatbot handle the user request correctly and complete the task? And when a task is reported as completed, is the chatbot hallucinating? Answering these questions is the critical part of analytics for managing the chatbot workflow. Finally, the value of the chatbot to the business should be calculated.
Engagement and retention rates
Engagement in a chatbot describes meaningful interaction, not simply opening the chat window. A conversation is counted as engaged only when it meets a predefined rule, such as containing at least one substantive question, because extra messages alone do not establish value. Active visitors follow the same logic and are defined as distinct identified users who sent at least one message or completed a defined interaction.
Four metrics describe engagement in practice. Conversation volume shows how often the chatbot is opened, queries per user show how often individuals return and drop off points show where users abandon a chat. Conversation length, measured in prompts and duration, completes the picture, although long conversations are usually read as a sign of overly complicated interactions rather than strong interest. Duration is only reliable when start and end events, an inactivity cutoff and a rule for open conversations are defined.
Retention is the hardest of these metrics to read, since a returning user may be satisfied or may be returning because the first conversation failed. Repeat contact about the same issue within a short window, such as seven days, is therefore a more informative signal. The absence of repeat contact still does not prove satisfaction, as the user may have moved to an untracked channel. For product level context, one month retention averages 46.9% across 83 SaaS companies, although this figure describes products rather than chatbots.
Fallback and goal completion
Fallback rate is the most direct measure of what the chatbot cannot handle. It is calculated as the number of eligible user messages that received a fallback response divided by all eligible user messages, with each message counted at most once. The trend is more informative than the level, since a rising rate suggests a topic the chatbot does not cover, while a single fallback can still occur in a conversation that ends successfully.
Goal completion measures whether the chatbot actually finishes the task it was asked to do. It is calculated as verified completions of a specified task divided by eligible attempts, and the word verified is essential. Containment and deflection are often used in its place, but deflection cannot distinguish users who were helped from users who gave up, so optimizing it can worsen outcomes. Successful containment is the stronger measure, dividing conversations with a confirmed resolution by all conversations handled and keeping handoffs in the denominator as failures.
A chatbot hallucinates about task completion when it reports a task as done, or answers with confidence, without the underlying action or knowledge supporting it. Typical cases are claims that are not grounded in the knowledge base and tool calls that select the wrong tool, fill arguments incorrectly or fail while the chatbot still confirms success. For this reason, hallucination rate and tool call success rate are tracked alongside goal completion.
Detection relies on checking responses against evidence rather than against reference answers, which rarely exist in production. Responses are scored against the source documents, and retrieval relevance is scored separately so that retriever failures can be distinguished from model failures. LLM judges reach over 80% agreement with human preferences but show position, verbosity and self enhancement biases, while word overlap metrics correlate weakly or not at all with human judgement. Automated resolution labels should therefore be checked against a manually reviewed sample, and scenarios should be run more than once, since in one voice agent benchmark a 95.12% per call completion rate dropped to 70.73% when all three runs had to pass.
The timing of escalations completes the picture. Early escalations point to routing problems, while late escalations show that the chatbot struggled for several turns before handing over.
Sales conversion and revenue tracking
Conversion is defined through verified outcomes rather than early signals. For lead generation, a qualified lead should be verified in the CRM, while a form submission or a link click is treated only as an earlier event. For SaaS products, activation and adoption lift indicate whether chatbot use is associated with activation or feature adoption, and sales chatbots can be linked to existing conversion tracking to count bookings or orders.
Revenue tracking follows the same caution. Orders from carts created by the agent can be matched back to conversations, but this attribution does not prove that the chat caused the purchase. On the efficiency side, containment can be converted into estimated savings, and each failure can be ranked by its cost in handoffs, repeat contacts or churn.
Cost per resolution connects spending with outcomes. It is calculated as the fully loaded cost, including model spend, tooling and human escalation time, divided by resolved conversations. The definition of a resolution changes the result considerably, as an illustrative allocation of $120 gives $10 per resolution across 12 classified resolutions but $15 across 8 verified ones.
A note on benchmarks
Few reliable benchmarks exist for chatbot metrics. There is no useful universal benchmark for deflection, and most published figures are vendor reported results for their own customers. A baseline built on the chatbot's own traffic is the more reliable reference.
Chatbot analytics platforms and dashboards
Chatbot analytics relies on two types of tools. Conversational analytics platforms track the user experience, such as resolution, fallback and satisfaction, while LLM observability tools such as Langfuse track model behavior, including traces, latency, token usage and cost.
Core features for conversational analytics
Conversational analytics starts with logging at two levels. Each message records the question, the response, the retrieval confidence, the source passages and the response time, while each conversation records turn count, duration, topic, sentiment, customer rating and any human takeover. Daily rollups per agent keep long term trends fast to query.
Dashboards present metrics such as completion rate, response time and escalation rate per agent and across the workspace, and every figure should link to the conversations behind it. Review is prioritized through a list of low confidence answers paired with the questions that caused them and a composite quality score that surfaces the weakest conversations. More advanced platforms also use LLMs and machine learning to find user clusters, unrecognized intents and systematic errors, which feeds knowledge base updates.
Chatbot data is most useful when combined with other company data through BI tools such as Tableau or through export to CSV, JSON or HTML reports. Privacy belongs among the core features, since conversation logs contain personal data. Typical controls are configurable retention windows, scheduled deletion, hashed IP addresses and aggregates that remain after transcripts are deleted.
Open source tools and Langfuse evaluation
Chatbots built on LLMs need visibility into what happens inside the model, and open source observability tools provide it. Langfuse is a representative example, described as an open source AI engineering platform that can be self hosted for privacy and compliance. Its tracing is based on OpenTelemetry, and each trace records the LLM calls, retrieval steps and tool invocations behind a response, while sessions group traces into full conversations with cost and token usage tracked per user.
Evaluation in Langfuse works by attaching scores to traces. Scores come from LLM as a judge evaluators, human annotation, custom code and user feedback, and dashboards show quality, cost and latency over time with alerts when a threshold is crossed. Prompt versioning and datasets close the loop, as problematic conversations are turned into test sets and prompt changes are checked against them before release.
FAQ
- Is a traditional chatbot the same as ChatGPT?
A traditional chatbot is not the same as ChatGPT, since a traditional chatbot follows predefined rules or intent scripts while ChatGPT generates its responses with a large language model. A traditional chatbot matches each request to a known intent and returns a fallback response when no match is found. ChatGPT can handle open ended questions, but it can also produce unsupported answers, which is why analytics for LLM based chatbots tracks hallucination rate in addition to fallback rate.
- What information should never be shared with an AI chatbot?
Information that should never be shared with an AI chatbot includes passwords, bank account and payment card numbers, government ID numbers and confidential business or medical data. Conversations with chatbots are usually logged and may be reviewed or analyzed, so anything typed into a chat should be treated as potentially stored.
- How does chatbot analytics optimize sales performance?
Chatbot analytics optimizes sales performance by linking conversations to verified outcomes such as qualified leads, bookings and orders. Leads are confirmed in the CRM and orders from carts created by the chatbot are matched back to the conversations behind them, which shows where the chatbot contributes to revenue. Cost per resolution and the savings from successful containment complete the picture, although attribution shows association rather than proof of cause.
- Why are open source platforms used for chatbot evaluation?
Open source platforms are used for chatbot evaluation because they can be self hosted, which keeps sensitive conversation data on internal infrastructure, and because their tracing and scoring can be inspected and extended. Langfuse, for example, builds on OpenTelemetry to reduce vendor lock in and combines traces, scores and test datasets in a single workflow.


