Evaluating enterprise AI vendors: ask for reproducible evidence

Start from the workflow

Feature names do not establish fitness for your organization. Define representative tasks and data boundaries, then give candidates the same controlled inputs and acceptance criteria. NIST offers a lifecycle risk frame; this editorial checklist does not certify or endorse any vendor.

Ask for quality and access evidence

Request answers grounded in current documents, rejection of obsolete policy and an explicit limitation when evidence is absent. Check real citations under the user's permissions. Test two roles with different access using synthetic or nonsensitive data. A prerecorded demo is not equivalent to reproducible execution against the agreed scenario.

For more on this topic, read How to evaluate AI-agent performance in the Liyan knowledge base.

Include operations and the exit path

Estimate licensing, infrastructure, ingestion, inference, maintenance and human review together. Ask what happens during model outages and queue saturation. Demonstrate export of relevant data and configuration, including format and migration limits. Test restoration rather than accepting the mere presence of a backup feature.

Record evidence and unresolved gaps

For each criterion, record the need, supplied evidence and remaining gap. Treat essential controls as eligibility gates rather than scores that attractive interfaces can offset. Document pilot scope, accountable owner and acceptance outcomes. Unverified claims should remain explicitly unverified until demonstrated.

Give candidates the same controlled test pack

Prepare synthetic current and obsolete policies, a table with units, and a role-restricted document. Include ordinary, unsupported and forbidden requests. Observe execution, setup effort and limitations. Record manual cleanup and custom tuning as implementation work. Allow comparable preparation and reserve fresh examples for the final assessment.

Turn security questions into bounded demonstrations

Run the same query under two roles, then revoke access and observe propagation. Restricted content must not leak through titles or suggested links. This is a scenario-specific check, not a complete security audit. Link processing destinations, retention and subcontractor statements to technical or contractual evidence. Include logs and backups. Leave unsupported claims explicitly unverified.

Cost the workload and demonstrate the exit

Specify active users, document volume, change frequency and accepted requests. Separate dated quotations from planning assumptions, and include ingestion, reprocessing, support and review. Export sample documents, metadata and transferable configuration and read them in a test destination. Record nonportable components and reconstruction work. Demonstrate restoration. Apply mandatory gates before weighted preferences and write a decision with evidence, gaps, owners and delivery criteria. Resolve close decisions through a focused test of the decisive gap.

A decision workshop with comparable evidence

Before demonstrations, agree mandatory conditions and preference weights with business, technical and operational owners. Give each candidate the same setup opportunity and controlled task pack. Preserve input versions, results, required preparation and unresolved limitations. Use fresh questions for the final comparison so a tuned demonstration does not stand in for everyday performance. Essential access controls should be assessed as gates before attractive interface features receive a score.

Observe restoration and data export rather than accepting feature labels. Read sample exported metadata and configuration in another environment, and record which components need rebuilding. Include initial conversion, ongoing changes, support and human review in a workload-based estimate. Keep dated vendor quotations separate from editorial planning assumptions. Run growth and failure scenarios because a small demonstration may omit the costs of retries, backlogs or large document updates.

For every claimed control or capability, write the requirement, supplied evidence, status and gap owner. Unsupported claims remain unverified rather than automatically passing or failing. End with a narrow proposed scope: the task, data, users, deliverables and acceptance procedure. Where candidates are close, identify the uncertain factor that can change the decision and test that factor directly. A longer feature list rarely resolves a concrete uncertainty about permissions, recovery or task quality. The useful output is a defensible choice and enforceable delivery criteria, including an exit path if the workflow later needs to move.