AI red teaming: practical safety and security testing for AI systems

What is AI red teaming?

Red teaming is a controlled attempt to find weaknesses before a real attacker does. In AI, scope can include prompts, data, tools, connectors, permissions and output.

The goal is not a dramatic break; it is a trackable control improvement.

What should be tested?

Prompt injection, data disclosure, permission bypasses, unsafe tool use and risky outputs should be prioritized by the real use case.

• Untrusted prompts and files
• Sensitive data and secrets
• Tool calls and external actions
• Roles and permission boundaries
• Incident logging and response

For more on this topic, read When a document gives instructions to an AI assistant in the Liyan knowledge base.

A safe red-teaming process

Define scope, authorization, test data, impact limits, evidence capture and stop criteria first. Testing should not expose real data or production services without need.

Findings should connect to an owner, severity, remediation, retest and acceptance date.

Why it must be continuous

A model, prompt, source, tool or policy change can alter the attack surface. Red teaming belongs in the release cycle alongside ongoing evaluations.

What are AI Evals? (Read more)

Build scenarios from real risk

Test scenarios should connect to the actual use case: which data is reachable, which tools the agent can use, and what the worst impact of a wrong decision would be. This makes remediation priorities clear.

Exercise malformed input, untrusted files, conflicting requests and attempts to cross role boundaries in a controlled environment. The goal is to capture impact and the required control, not to recreate harm.

From finding to verifiable remediation

Record severity, evidence, remediation owner, target date and retest criterion for every finding. A vague statement that a model is safer is not a measurable result.

After a fix, rerun the same and adjacent scenarios. A source, prompt or tool change can reactivate an old path, so testing belongs in release work.

Scenario: testing a knowledge-connected agent

One test can introduce an untrusted document that asks the agent to ignore earlier instructions or expose confidential data. The expected behavior is explicit: document text must not become policy.

Another test can request a tool outside the user's role. The trace should show that permission controls stopped the request and that sensitive data did not appear in output or logs.

Each result connects to risk severity and an remediation owner. Rerun the scenarios after changing a prompt, connector or model so a fix remains verifiable.