An AI failure can look dramatic and still be difficult to reproduce. Model behavior changes with prompts, system instructions, retrieval data, tool permissions, model versions and sampling settings. A useful red-team finding preserves enough of that environment to show what happened, why it matters and whether a repair worked.

Define the system, not just the model

The test scope should identify the complete application path: model and version, system prompt, retrieval sources, safety layers, connected tools, identity context, data boundaries, output handling and human review. A finding against a base model may not describe the deployed application. The reverse is also true: secure model behavior can be undermined by permissive tools or unsafe retrieval.

Write the threat model in business terms. Relevant outcomes may include unauthorized data access, unintended action, misleading advice, insecure code, policy evasion, sensitive-data disclosure or manipulation of a downstream decision.

Capture the conditions of each test

For every material result, retain the prompt sequence, system state, model identifier, parameters, connected data, account role, time, output and any tool call. If a test relies on many stochastic attempts, report the number of attempts and the success rate. One surprising output may be worth investigation, but it should not be presented as a stable exploit without supporting evidence.

  • Use unique test cases with a clear objective and expected safe behavior.
  • Keep baseline, adversarial and control runs.
  • Record data and prompt transformations performed by middleware.
  • Separate policy failure from exploitable business impact.
  • Preserve failed attempts when they affect likelihood estimates.

Test authorization boundaries

Tool-enabled systems deserve special attention. Can the model invoke a function the user could not call directly? Can retrieved content influence tool arguments? Are high-impact actions confirmed outside the model conversation? Do logs show the initiating user, generated request, approval and result?

Indirect prompt injection should be tested through the places untrusted content enters the system, such as files, websites, tickets or knowledge bases. The test should show whether hostile instructions can cross a trust boundary, not merely that the model can repeat adversarial text.

Write findings for repair and retest

A strong finding includes the preconditions, exact steps, observed behavior, affected assets, credible impact, likelihood, existing controls and a proposed control objective. Attach a reproducibility bundle that another tester can run without guessing at hidden state.

Repairs may belong outside the model. Least-privilege tools, input isolation, output validation, deterministic policy checks, approval gates, monitoring and retrieval controls can be more dependable than another instruction added to a system prompt.

Retest against the same evidence

  1. Repeat the original test under the recorded conditions.
  2. Run neighboring variations to detect a narrow patch.
  3. Verify the intended function still works for legitimate users.
  4. Check whether the repair moved risk to another tool or channel.
  5. Record model, application and policy versions in the retest result.

The NIST Generative AI Profile applies the AI Risk Management Framework’s Govern, Map, Measure and Manage functions to generative systems. It treats red teaming as one part of broader test, evaluation, verification and validation work, which is the right frame for turning individual tests into managed evidence.

Testing limit: Red teaming samples behavior under defined conditions. It cannot prove that a probabilistic system is safe in every context, and authorization must cover the connected systems, data and actions being tested.