Agent evaluation
A Sankey diagram of an AI support agent's contact journeys, and the bars it had to clear before launch.
Recreated on dummy data. Every name, figure and date in the recreation is invented.
Result: The evaluation gated the support agent’s rollout. It held at 10% until 6 fixes lifted correct escalations from 65% to 90%, then shipped to all customers at 95% and cut outsourced support volume 60%.
Problem: The agent project fell behind schedule, and the team had no clear way to measure whether the agent worked.
What I built:
- A Sankey diagram of the customer journey from existing conversation logs: drop-offs, abandonments, correct and incorrect closures, escalations, and contacts that should have been escalated but were not.
- Two evaluation metrics with a bar each: whether the agent resolved the issue without escalating, and whether each ticket it created was needed.
- A rollout gate: keep refining until both metrics clear the bar, then monitor before the next stage.
- A reset of the project’s planning, bandwidth and hiring once it slipped.
How I worked with AI: I recreated the evaluation here with Claude Code in VS Code, from my own description, on dummy data.
Decisions and trade-offs:
- I measured the agent on existing conversation logs instead of building a new test set first. The cost was judging only the conversations customers had already had.
- I held the rollout until both metrics cleared the bar. The cost was a later launch for most customers.
What changed: Correct escalations from 65% to 90% across 6 fixes. Rollout from 10% to all customers at 95%. Outsourced support volume down 60%.
Links: The recreation is below. Dummy data generator ยท Dummy data