The existing matcher returned no internal location code for one distribution-center record. The AI operations review system checked the current reference list, found one active facility with the same company identity and an exact address match, and returned the supported code. The hidden analyst answer later confirmed it.
That was the kind of case I wanted the system to remove from a manual queue. The harder test was whether it would refuse cases where the evidence looked convincing but did not prove the business decision.
In an accepted 337-case blind prototype evaluation, the agent made 62 verified correct automatic decisions—about 18% of the queue—and retained 275 for human review. There were no confirmed wrong automatic outcomes in that safety-gated snapshot. That is not a production accuracy claim, and it does not yet prove that 18% of the human workload disappeared.
The exception queue was the bottleneck
The company already had a rules-based process that handled straightforward records. The difficult remainder involved incomplete names, conflicting geography, historical aliases, changing reference data, and business relationships that were not written down in one authoritative place.
Replacing the existing matcher would have discarded the part that already worked. I kept it as the first pass and sent only uncertain records into the evidence layer. The AI could confirm the matcher's suggestion, recover a supported code the matcher missed, confirm that no code should be returned, or abstain and explain what a person still needed to decide.
I spent roughly 30 hours over two weeks building and testing the prototype. Most of that time did not go into prompt writing. It went into examining failures, tightening evidence boundaries, and deciding what the system was not allowed to infer.
That distinction matters because an exception queue is not one generic fuzzy-matching problem. Some records are missing evidence. Others contain strong identity evidence but still depend on an undocumented routing rule. Treating those cases as equivalent produces a confident answer where the business has not actually granted authority.
Bounding the AI operations review system
The review layer combined the existing result, current approved reference data, approved historical knowledge when applicable, structured facility relationships, fuzzy retrieval, semantic retrieval, and bounded external research.
Those sources did not vote on the answer. Fuzzy and semantic search could retrieve a candidate or a relevant explanation. External research could support company or facility identity. None of them could independently authorize an internal operational decision.
Every automatic outcome still had to agree with current reference data and an applicable approved rule or facility-specific mapping. When evidence conflicted, the system narrowed its behavior instead of hiding the conflict behind one confidence score.
That control boundary also follows the governance direction in the NIST AI Risk Management Framework: model output still needs accountable controls around how it is interpreted and used.
That follows the same boundary-first approach I used when I moved MCP authentication and authorization into AWS. The model can help assemble evidence, but the highest-risk decision belongs in a deterministic control outside the model's reasoning.
Testing the AI operations review system
The accepted demo snapshot contained 337 blind cases from new-customer evaluations. The final results were:
- 62 verified correct automatic outcomes;
- 275 records retained for human review; and
- no confirmed wrong automatic outcomes in this accepted safety-gated snapshot.
The 62 automatic outcomes included 17 confirmations of a code already suggested by the deterministic matcher, 42 correct recoveries where the matcher had not supplied the final correct code, and three correct confirmations that no code should be returned.
The important number is not only 62. It is also 275. The system did not need to produce an answer for every record to be useful. It needed to automate the supported cases and preserve human authority everywhere else.
Earlier experiments used different knowledge conditions and produced higher automation rates. I am not combining those figures with the blind result because they are not directly comparable. A near-repeat test with approved history answers a different question from a new-customer blind evaluation.
Retrieval is not authority
The most useful failure happened when the AI treated a similar historical relationship as authorization for a new record. The note was relevant, but it came from a different customer context and did not prove that the same routing relationship governed the new case.
I added a stricter rule: historical notes could support an investigation, but they could not authorize a different code unless an approved facility-specific mapping or applicable business rule covered the new record.
That rule changed how I evaluated the system. A candidate could be geographically close, share a name, or even point to the correct physical facility without proving the required business destination. Facility identity and operational routing were separate questions.
One refusal made that boundary visible. The matcher suggested an active facility far from the incoming record. The AI found stronger exact-address evidence for another facility and a plausible local business relationship, but no approved rule established which relationship controlled the final code. It sent the case to a person and named the missing decision instead of choosing the most plausible option.
The analyst later selected the local relationship. The AI had found useful evidence. It had also correctly recognized that it lacked authority.
Human review stayed explicit
The first useful interface was not an autonomous pipeline. It was a review screen that brought the incoming record, deterministic result, current reference candidates, approved history, AI-prepared evidence, conflicts, and abstention reason into one place.
The analyst could resolve the current record without automatically creating a permanent rule. If the explanation contained reusable knowledge, the AI proposed a structured interpretation—such as an alias, facility relationship, routing rule, scope, or limitation—and a person approved, edited, or rejected it separately.
Nothing became active knowledge merely because the model suggested it. That separation allowed the workflow to improve without turning every one-off correction into policy.
This is the same reason I evaluate AI systems as operating workflows rather than model demos. In another agent validation project, the useful result was not simply that the model could call tools. It was that the prototype exposed the controls and unanswered questions before a larger production investment.
The next test is real work removed
The prototype established a defensible gross result: 62 supported automatic outcomes in a 337-case blind snapshot. It has not yet established the net reduction in human work.
If operations must fully recheck every automatic outcome, the apparent 18% queue reduction may produce very little actual time savings. The demo therefore needs to answer a more practical set of questions: Can a supported outcome bypass full review? Does the prepared evidence shorten the cases that still need a person? Do reviewers understand and trust the abstention reasons? Does the saved effort exceed the model and maintenance cost?
Until those answers exist, I will not publish a labor-savings or ROI claim. The honest result is narrower: the agent identified a verified set of routine decisions while leaving unresolved policy and routing conflicts with people.
What the results do—and don’t—prove
Did the AI replace the existing matcher?
No. The deterministic process remained the first pass. The AI concentrated on uncertain records that needed additional evidence or a prepared review.
Does 18% mean production accuracy?
No. It is the supported-outcome rate from one accepted 337-case blind prototype snapshot. It should not be generalized to future files or production performance.
What happens when the evidence conflicts?
The system returns human review with the relevant candidates, evidence, conflict, and abstention reason instead of guessing.
If you are adding AI to an existing operations workflow, start by defining the refusal path. An AI operations review system that cannot explain when it lacks authority is not ready to expand the automation boundary.
