Field Notes · Elle Anderson · andersonco.ukField note · AI-era practice
Product Metrics After the LLM: What to Count Now
The dashboards most product teams run were designed for a world where software did what it was told. AI products do not. They answer differently each time, they are wrong in ways that look right, and their value depends on a human trusting them. Most of the numbers on the dashboard have not kept pace. Here is what has changed, what the industry is counting instead, and what I would put on the wall.
What the SaaS Era Taught Us to Count
Daily and monthly active users. Sessions. Feature adoption. Net Promoter Score. Conversion through a funnel. These were good metrics for software with deterministic behaviour, because the only variable was whether the customer arrived and whether the funnel converted them. The product either did the job or it did not, and you could tell from the logs.
AI products broke two assumptions underneath that dashboard. First, the product's behaviour is now a variable: the same request can succeed on Monday and fail on Tuesday, and the failure can be fluent enough that nobody notices. Second, value now depends on trust, which is not an event you can log. A customer who received a wrong answer last week and quietly stopped relying on the feature still shows up as active.
The New Vanity Measures
The AI era did not fix the vanity problem. It added a fresh set. Prompts sent, tokens served, "AI interactions", suggestion acceptance rates, deflection rates, percentage of users who have tried the AI feature, accuracy on the eval. Every one of these is easy to collect, goes up and to the right in the first quarter, and tells you almost nothing about whether the thing is working. They measure activity on the way to an outcome, and in an AI product the gap between activity and outcome is exactly where the risk lives.
Here is how that plays out in the products everyone is watching.
| WHERE YOU SEE IT | THE VANITY NUMBER | WHY IT LIES | COUNT THIS INSTEAD |
|---|---|---|---|
| Coding assistants GitHub Copilot, Cursor and the rest | Suggestion acceptance rate, lines of AI-written code | Accepted is not kept. Code that is accepted, then edited away, then reverted in review counts the same as code that shipped. More lines is often more debt. | Share of AI-assisted changes still in the codebase after thirty days. Review cycle time. Defect rate of AI-assisted changes against the baseline. |
| Customer service agents The Klarna-style assistant every bank now has | Deflection rate, conversations handled by AI | Deflected is not resolved. A customer who gives up, phones instead or churns was "handled". Klarna publicly celebrated the volume its assistant took, then publicly brought humans back into the loop when quality told a different story. | First-contact resolution as the customer experiences it. Repeat contact within seven days on the same issue. Satisfaction on AI-handled contacts versus human-handled, same issue type. |
| Assistants in a workspace Notion AI, Microsoft Copilot, Slack AI | AI feature adoption, prompts per user, "percentage of users who tried it" | Everything new gets tried. The number that matters is who is still using it in month four, for a task it was built for, and whether it displaced a worse way of doing the same thing. | Week-sixteen retained use for the intended job. Time to complete the task with and without the feature, measured on real work. Seats that would be cancelled if the feature disappeared. |
| Search and answer engines AI overviews in search, answer boxes in apps | Impressions, answers shown, queries served | An answer shown is not an answer trusted. If users read the summary and then click through to check it, the feature is a cost with a reassurance problem. If they stop clicking, you have moved the trust risk onto yourself. | Accuracy as experienced on a graded weekly sample, weighted by what a wrong answer costs. The moment after: verify, abandon or act. Complaints and corrections per thousand answers. |
| Recommendation and personalisation Streaming, retail, feeds | Click-through on recommendations, engagement minutes | A model can raise clicks by recommending the obvious or the outrageous. Both show up as engagement. Neither builds the thing you actually sell, which is a customer who comes back next month. | Next-month retention of customers exposed to the model versus a holdback. Catalogue breadth consumed. Regret signals: skips, returns, unsubscribes within the session. |
| Any LLM feature Including the one on your roadmap | Accuracy on the eval, tokens served, cost per interaction | The eval was built by the people who built the model, on the cases they imagined. Tokens measure activity, not outcome. Cost per interaction hides cost per result. | Cost per successful outcome, and its curve at ten times the volume. The kill criterion, written before launch, with a name next to it. |
Two of those rows deserve a note. The Klarna story is the clearest public lesson of the LLM era so far: a company reported, loudly, how much of its customer service its assistant was handling, and later said, just as publicly, that it was bringing people back into the loop because quality mattered more than the volume. The metric they led with was true and beside the point. And the coding assistant story is the one most engineering leaders are living right now. Acceptance rate is the number the vendor gives you. Retained code and defect rate are the numbers your business runs on, and they are not the same number.
Best Practice: The Measures That Survive Contact with a Customer
Strip the dashboard back to the measures that would change a decision, and in an AI product there are six.
1. Task success as experienced. Not accuracy on the eval. Sample real interactions every week, have a human grade them against what the customer was trying to do, weight by what a wrong answer costs, and track the trend. It is slower than an automated score. It is also true.
2. The cost of a wrong answer, on the dashboard. Every AI feature has one and it is almost never shown. Estimate it, even roughly, multiply by the error rate as experienced, and put the figure beside the savings in the business case.
3. The moment that follows. What the customer does immediately after a wrong or low-confidence answer: retry, verify, abandon, work around, complain or quietly stop trusting. Instrument that one event and you have the most informative signal in the product.
4. Retained use for the intended job. Who used it in month one and is still using it in month four, for the task it was built for, split by whether they met a wrong answer in week one. On every AI product I have measured, that split explains most of the retention curve.
5. Cost per successful outcome, and its curve. Cost per interaction hides the real number. Cost per result that the customer accepted, at today's volume and at ten times today's volume, is what the finance director will ask for in a year.
6. The kill criterion. The measured result, by a date, that switches the feature off, and the name of the person with the authority to pull it. If that sentence cannot be written before launch, the dashboard is collecting evidence for a decision nobody intends to make.
How to Run It
Keep the SaaS metrics. They still tell you whether people are coming. Then, for every AI feature, add a second row of exactly these six, owned by the product lead and reviewed in the same meeting as revenue. Grade a sample by hand every week; forty interactions is enough to see a trend. Put the wrong-answer cost in pounds, not percentages, because pounds end arguments. And when the vendor's number and your number disagree, your number is the one that is true.
The organisations shipping AI value in two years' time will not be the ones with the highest adoption charts this year. They will be the ones who knew, every week, whether the thing was working, and had the nerve to switch off the ones that were not.