Three Signs Your AI Pilot Is Measuring the Wrong Thing

If your AI pilot report leads with how many messages the agent sent, how many tasks it touched or how many hours it saved, the pilot is measuring activity rather than results. Those three numbers all go up when the tool is working and also when it is quietly wasting everybody's time, which makes them useless for deciding anything.

The test for any pilot metric: could this number rise while the work got worse? If yes, it is not a result. It is activity.

Sign one: you are counting messages, drafts or outputs

Volume is the easiest thing to count and the least informative. An agent that produces four hundred drafts a month has produced four hundred things for somebody to read. Whether that helped depends entirely on how many survived review, and volume does not tell you.

Worse, volume rewards the wrong behaviour. A tool tuned to produce more suggestions will score better on this metric while making the reviewer's day longer.

What to count instead: outputs a person approved and used, as a share of outputs produced. If nine in ten drafts get rewritten, you have a very expensive way of generating a starting point.

Sign two: you are counting tasks touched

This one hides inside project reporting. The agent updated two hundred tasks, flagged sixty risks, commented on ninety items. It looks like coverage.

The problem is that touching a task is not the same as improving it. An agent that adds a comment to every item on the board has touched everything and helped with nothing. A flag that turns out to be wrong costs more than no flag at all, because somebody investigated it, which is clearest once wrong actions are priced in hours to correct.

What to count instead: how many of the flags were real. Take a sample of the risks the agent raised and have the person who owns that work say whether each one mattered. A low hit rate is not a small problem. It teaches the team to ignore flags, which removes the value of the ones that were right.

Sign three: you are counting hours saved

Hours saved is the number executives ask for and the number hardest to trust, because it is almost always calculated rather than observed.

The usual method is to multiply outputs by an assumed time per output. That assumption came from somewhere, usually a guess or a vendor. Then nobody subtracts the setup time, the time spent fixing outputs that were wrong in an interesting way, or the review time nobody budgeted for.

There is a simpler failure too. Hours saved only counts if the hours went somewhere. If the person now spends the freed time on more of the same work, you have capacity. If they spend it waiting, you have nothing.

What to count instead: the end to end time for the whole workflow, measured, from the moment the work arrives to the moment it is finished and approved. That figure includes review, includes rework, and cannot be inflated by assumptions.

What good pilot metrics look like

Three properties make a metric worth reporting.

• It can go down when the work gets worse, not just up when the tool gets used
• It is observed rather than calculated from an assumed baseline
• The person who does the work agrees it reflects their day

A few that usually pass:

• Approval rate: outputs accepted with no or light editing, out of all outputs produced
• End to end cycle time for the workflow, before and during the pilot
• Errors that reached a customer, counted the same way before and after
• Backlog size in the queue the agent is meant to help with
• Reviewer minutes per approved output

The awkward one nobody reports

Ask the reviewers what they rejected this week. If the answer is regularly nothing, either the agent is perfect or nobody is checking. Assume the second.

A pilot where rejection rate is zero and approval is instant is not a success. It is a pilot where the approval step has become a formality, which matters especially in systems built so a human approves agent output before it ships. The control only works if somebody is using it.

Fixing a pilot mid flight

If you are three weeks into a pilot measuring the wrong thing, you do not have to start again. Keep the setup, change the log.

Ask the reviewer to record one line per output: approved, edited, or rejected, with a few words on why for anything rejected. Two weeks of that log will tell you more than three months of dashboards, because it is the only record that captures whether the work was any good.

Then compare it against how the same workflow ran before. If you did not measure the before, measure it now by hand for a week, using the same handful of numbers you would capture ahead of a rollout. An imperfect baseline beats an imagined one.