From AI pilot to production: evidence beyond a convincing demo

Define the decision before the demo

A convincing answer establishes possibility, not day-to-day dependability. Name the workflow and user group, then measure the existing effort and correction burden. NIST provides a risk-management frame; the rollout process below is an editorial implementation pattern, not a universal certification threshold.

An editorial HR-assistant pilot

Restrict the trial to approved policies and named users. Include normal, ambiguous, out-of-scope and forbidden-access questions. Let the content owner label reference answers and correct documents. Measure time to an accepted answer, citation correctness and manual rework together. Fast delivery of obsolete policy is still a failure.

Make production readiness explicit

Assign a service owner, knowledge owner and incident path. Rehearse recovery, rollback and test-data removal separately from production. Minimize sensitive logs. Define stop criteria before rollout, including unauthorized disclosure or unapproved actions. These cases require a different response from ordinary satisfaction issues.

For more on this topic, read Building AI agents for production in the Liyan knowledge base.

Roll out to a limited group

Record model, prompt, index and policy versions. Classify feedback by data, retrieval, generation and interface problems. Avoid changing all components at once. Compare cost per successful task and quality against the baseline. Expansion, redesign or a justified stop can each be a useful outcome of a well-designed pilot.

Bound a leave-policy pilot

Start with finding the current policy clause and drafting a response for a reviewer. Exclude individual entitlement decisions and personnel-record updates from the initial scope. Measure the existing workflow from source discovery through accepted response, including corrections and escalation. Compare like-for-like completed work rather than model generation time against the entire manual process.

Write acceptance and stop conditions early

Label ordinary, exceptional, unsupported and forbidden-access cases with domain-expert reference answers. Reserve unseen examples for final evaluation. Agree context-specific quality targets and hard stop conditions before the demo. A prohibited disclosure can block rollout despite good average performance. The acceptance sheet should support a concrete release, repair or stop decision.

Make operational ownership part of the deliverable

Assign content and service owners, issue handling and a usable manual fallback. Record model, prompt, retrieval and corpus versions. Rehearse interruption and rollback before a small-user rollout. During operation, track accepted responses, material corrections, full task time and permission failures alongside usage. Investigate repeated questioning and failed sessions. Expand only when evidence supports it; conflicting documents may require content repair, and low-value workflows may deserve a narrower scope or closure.

Build a release packet that operations can use

For the leave-policy example, document authorized users, approved sources, excluded actions and the person responsible for accepting answers. Label reference cases before the demonstration. Keep unsupported questions and restricted-source requests in the test set even if most daily traffic is straightforward. Agree which failures demand a stop and which can enter a repair queue. A single overall percentage cannot express these different consequences.

Prepare a brief user guide showing how to inspect a citation, recognize a limited answer and report a contradiction. Establish the operational response to a missing source, model outage or backlog. Rehearse the manual fallback and decide who can roll back a release. Preserve the relevant model, prompt, retrieval and corpus versions so an observed regression can be tied to a change. A code handover without these decisions leaves important parts of production ownership unresolved.

During the initial rollout, compare complete accepted responses with the manual baseline. Include review and correction time. Investigate repeated questions, since higher message volume may signal difficulty reaching a usable answer. Classify failures into source conflict, retrieval, generation and interface comprehension. Choose the next change from that diagnosis: repair inconsistent policy, improve evidence display or narrow the supported scope. If the task does not create sufficient value, stopping or redesigning the pilot is a valid evidence-based outcome. Production readiness is the ability to deliver and operate the agreed workflow, not the presence of an impressive demonstration.