Field Notes · Elle Anderson · andersonco.ukField note · AI-era practice
Measuring AI Outcomes: What to Count
Ask a leadership team how their AI is performing and you will usually get one of two numbers: an accuracy score from the eval, or an adoption figure from the launch month. Neither tells you whether the thing is working. Here is what I count instead, learned on products that had to earn their keep before AI was fashionable.
The Two Numbers That Are Usually Reported, and Why They Mislead
Accuracy on the eval is a lab result. It tells you how the model behaves on a set of examples somebody chose, weighted however they chose to weight them. It says nothing about what your customers actually ask, how often the model is wrong on the questions that matter most, or what a wrong answer costs them.
Launch-month adoption is a novelty curve. Everything new gets tried. The number that matters is who is still using it in month four, and whether they are using it for the purpose you built it to serve.
Both numbers are easy to collect and both make a slide look good. That is why they get reported, and why most AI measurement is a kind of theatre.
1. Accuracy as Experienced
Take the real distribution of what customers ask, weight each type of question by what it costs the customer when the answer is wrong, and measure against that. On the computer vision product I was accountable for, we found the eval was proud of cases customers rarely hit and quiet about the one case that drove most of the complaints. Once product co-owned the eval with engineering, the number started to mean something.
The practical version: sample real interactions weekly, have a human grade them against the customer's outcome, and track the trend. It is slower than an automated score. It is also true.
2. The Cost of a Wrong Answer
Every AI feature has a wrong-answer cost, and it is almost never on the dashboard. A mis-identified part meant a wrong order, a return and a support ticket. A hallucinated figure in a financial summary means something worse. Put a number on it, even a rough one, and multiply by the error rate as experienced. That is the figure that belongs next to the cost savings in the business case, and it usually changes the conversation.
3. The Moment After
What does the customer do immediately after a wrong or low-confidence answer? Retry, abandon, work around, complain or quietly stop trusting the product. Instrument that moment. It is the single most informative event in an AI product, because it tells you whether your failure states are doing their job. A feature with a good moment after can survive a mediocre model. A feature with a bad one cannot survive a good model.
4. Retained Use for the Intended Job
Not adoption. Not sessions. The share of customers who used it in month one and are still using it in month four, for the task you designed it to do. Split that by whether they experienced a wrong answer in the first week. On every AI product I have measured, that one split explains most of the retention curve, and it tells you where to spend the next quarter.
5. Unit Economics at Ten Times the Volume
What one interaction costs today is interesting. What the curve looks like at ten times the volume is the number the CFO will ask for in a year, and the number that decides whether the feature survives being successful. Measure it from launch, and put the cost per successful outcome, not per interaction, on the dashboard.
The Measure That Sits Over All of Them
Kill criteria. The measured result, by a date, that switches the feature off, and the name of the person with the authority to pull it. If you cannot write that sentence before launch, you are not measuring outcomes. You are collecting evidence for a decision nobody intends to make.
None of this is exotic. It is the discipline that unfashionable machine learning forced on us, because nobody was going to keep a product that did not work. The gold rush lets teams skip it. The teams that don't are the ones whose AI will still be switched on in two years.