How to Test an Agent Before It Touches Live Records

You have an agent that is meant to update project records, and nobody wants to be the person who lets it loose on the real database first. Fair enough.

The method for testing an AI agent that works is a shadow run: give it real live inputs, take away its write access, and compare what it wanted to do against what actually happened. Agree what counts as a pass before you look at a single result, because deciding afterwards is how everything gets approved.

Why a sandbox with fake data proves nothing

The usual approach is a test environment with sample records. It catches crashes and it misses everything else.

Made up data is tidy. Real data has a project with no owner, a client whose name is spelled three ways, a task closed in 2023 that someone reopened last week, and a field somebody has been using for a purpose it was not designed for. Those are the cases where an agent does something surprising, and they only exist in the live system.

So use the live system. Just do not let the agent write to it.

Setting up the shadow run

The shape is simple.

• The agent reads live records, exactly as it would in production.
• It receives real requests, either mirrored from the live queue or fed from a recent backlog.
• Every action it wants to take is recorded rather than executed. Not a summary of the action. The actual change: this field, this old value, this new value.
• Meanwhile, humans keep doing the work as normal, and their changes are recorded too.

You now have two parallel records of the same period. One is what happened. One is what would have happened.

Run it long enough to cover a normal cycle of your work, including whatever weekly or monthly rhythm your team has. A run that only sees quiet Tuesdays tells you about quiet Tuesdays.

Agree the pass criteria first

This is the part people skip, and it is the part that matters.

Write down, before the run starts, what result would make you say yes. Then write down what result would make you say no. Both, in numbers or in specific described cases, and signed off by whoever owns the process. Settling it now saves a harder conversation later about who is accountable when approved output turns out wrong.

If you do not do this, here is what happens. The results come back, they are mixed, everyone looks at them, and a discussion begins about whether the disagreements were really errors. Some of them were not. Most of them were. The discussion resolves in favour of shipping, because a lot of work has already gone in and nobody wants to say so.

Useful criteria to set in advance:

• The rate of disagreement with human action that you can accept, and separately the rate you will treat as a failure.
• Which categories of error are disqualifying regardless of rate. Anything touching money, anything touching a person's record, and anything on the list of what an agent should never be allowed to email.
• The proportion of cases the agent should have escalated rather than acted on.
• What happens on a fail. Fix and rerun, or stop.

Deciding the disqualifying list in advance is the most valuable part. It is very hard to argue after the fact that a single bad case matters when the aggregate numbers look fine.

Reading the disagreements

Where the agent and the human differed, sort the cases into three piles.

The agent was wrong. Obvious, and useful. Look for a pattern rather than counting.

The human was wrong. This happens more than teams expect and it is genuinely valuable information about the process, separate from the agent question.

Both were defensible. The agent picked a different reasonable option. These are the interesting ones, because they tell you the task has judgement in it that nobody has written down. That is a decision for the process owner, not a bug to fix.

What a shadow run does not test

Be honest about the gaps.

It does not test load. It does not test what happens when the agent's actions change the state that later actions read, because in shadow mode they never do. It does not test how people behave once the agent is live, and people behave differently: they stop double checking, they start phrasing requests for the machine, they route odd cases to it because it is faster.

Those need a limited live phase with a small scope and a human approving every write, following one of the approval patterns that hold up in regulated work. Shadow first, narrow live second, wider later.

Where the record helps

SicherOne holds project, HR and agent activity against one set of records, so a shadow comparison has the human action and the proposed action side by side rather than in two exports that have to be reconciled by hand. A human approves agent output before it ships, so the narrow live phase does not require building an approval step from scratch.

The habit worth keeping after the test is the recording. The comparison that told you whether to launch is the same comparison that tells you, six months later, whether anything has drifted.