AI Operations Review System Case Study

The existing rules stayed in charge of the easy work

Direct comparison with the final human-reviewed answers showed that the existing process was right on 3,509 of 4,175 baseline records and disagreed on 666. The old system was already doing most of the work.

The risk was treating the remainder as one generic fuzzy-matching problem. Some records lacked a supported destination. Some had a technically plausible candidate that the business had intentionally not approved. Others represented the same physical facility under different names, brands, or customer systems.

I kept the deterministic matcher as the first pass. Clear results stayed deterministic. The AI layer only received records that needed more context, investigation, or explanation.

Retrieval found candidates; policy authorized decisions

The review layer combined the existing result, current approved reference data, exact approved history when the test allowed it, structured relationships, fuzzy name and geography retrieval, semantically relevant analyst explanations, and controlled external research when identity remained uncertain.

Those sources did not vote on an answer. Fuzzy and semantic retrieval could find a candidate or explanation. External research could support company or facility identity. None could independently authorize an internal operational decision.

Every automatic result still had to agree with current reference data, the deterministic rules, and any approved historical decision. Conflicting evidence reduced automation instead of being hidden behind one confidence number.

The review path was part of the architecture

The early interface centered on one review screen rather than a polished autonomous workflow. It brought the incoming data, current deterministic result, approved history, reference candidates, AI-prepared research, conflicts, and abstention reason into one place.

The analyst could resolve the current record without automatically turning that explanation into a permanent rule. Reusable aliases, facility relationships, and decision notes needed separate approval and provenance so a correction could be audited or retired later.

The system therefore improved two different things. It reduced reviews when evidence was uniquely sufficient, and it prepared better reviews when a person still had to decide.

Blind tests exposed missing authority, not just model errors

One blind evaluation held the final answers away from the agent and tested 37 lower-confidence records from a different customer file. The corrected result contained nine exact automatic decisions, 26 safe review decisions, two unresolved policy cases, and zero wrong automatic decisions.

The run exposed a dangerous pattern: approved history from one customer could identify a related company or facility but still recommend the wrong operational destination for another customer. I added a narrow safeguard so history could confirm the deterministic candidate but could not silently replace it.

A later cross-customer sample found another boundary. Exact address evidence could identify the physical facility while still conflicting with an older human disposition. I separated those disagreements into a policy-and-data queue because the evidence explained the facility without proving what the business wanted returned.

The controlled result is evidence, not production accuracy

The 180 automatic decisions and 51 safe reviews came from a controlled set of modified, previously approved cases. The source record ranked first for all 231 inputs, but only 20 remained exact after normalization. The system still abstained when the modified input conflicted with current evidence.

This proves that the retrieval and decision gates survived controlled spelling, punctuation, abbreviation, and geography changes. It does not prove production accuracy across every customer, a permanent automation rate, realized labor savings, or a finished production analyst workflow.

At the latest verified checkpoint, the system remained a proof of concept and early workflow preview. The approval interactions demonstrated the intended process; they did not yet write directly into production knowledge.

The architecture applies beyond record matching

The reusable pattern is to keep the deterministic system for decisions it already handles, send only unresolved records into the evidence layer, separate candidate retrieval from authorization, make abstention a defined result, and approve reusable knowledge separately from the current decision.

If an operations workflow cannot explain why a result is authorized, it should not hide that uncertainty behind a model score. Build the review path before expanding the automation boundary.

Frequently asked questions

Did AI replace the existing deterministic matcher?

No. The existing process remained the first pass. AI concentrated on unresolved or conflicting records that needed additional evidence or review preparation.

Does 77.9% represent production accuracy?

No. It is the safe-automation rate from one controlled 231-case reuse evaluation. It should not be generalized to future customer files or production performance.

What happens when the evidence conflicts?

The system returns human review with the relevant candidates, evidence, conflicts, and abstention reason instead of guessing.

Can external research authorize the final operational decision?

No. Research can support identity or address evidence, but only current internal data, approved policy, and authorized human decisions can approve the result.