Technical field guide
The sections below preserve the service-specific depth behind AI Security Testing, edited for the current national practice and its documented engagement model. Methods are selected for the source, authorization, system state and assigned specialty. No single tool or artifact establishes a conclusion, and legal, regulatory or certification decisions remain with the responsible authority.
AI Security Risks in Production Deployments
The deployment of large language model applications in production environments has introduced security risks that differ fundamentally from those addressed by traditional application security programs. An LLM application is not merely software with inputs and outputs. It is a probabilistic reasoning system whose behavior depends on training data, system prompt configuration, retrieval context, tool integrations, and the cumulative effect of conversation history. Adversaries who understand these dependencies can manipulate model behavior in ways that bypass conventional access controls, extract sensitive information, and in agentic deployments, trigger unauthorized actions against connected systems.
The attack surface of an LLM application includes every channel through which external content reaches the model. Direct user input is the most visible channel, but it is far from the only one. Retrieval-augmented generation (RAG) systems ingest documents, database records, and web content that may contain adversarially crafted instructions. Autonomous agents that browse the web, read email, or process uploaded files are exposed to indirect prompt injection through any content they consume. Multi-model pipelines introduce additional trust boundaries where one model's output becomes another's input, creating potential for cross-model injection and output handling failures.
The consequences of AI security failures range from confidentiality violations to operational disruption. System prompt extraction exposes the instructions and guardrails an organization has invested in building. Data extraction through carefully constructed prompts can produce customer records, internal documents, or credentials stored in the model's context window or connected data stores. In agentic deployments with tool access, a compromised model may send unauthorized API calls, modify data, or exfiltrate information through channels the application was designed to use for legitimate purposes. These are not theoretical risks: documented incidents have demonstrated prompt injection attacks against production AI assistants, customer service chatbots, and coding tools used in enterprise environments.
The testing methodology is built specifically for LLM systems and the threat surface they expose. For organizations designing security architecture into AI deployments from the ground up, see GDF's AI security architecture services.
OWASP Top 10 for LLM Applications
The OWASP Top 10 for LLM Applications (2025) provides one useful risk taxonomy. Each category represents a distinct class of risk examined in client AI deployments.
LLM01: Prompt Injection. A central AI vulnerability class, prompt injection occurs when adversarial input causes the model to override its instructions or safety measures. GDF tests for direct injection through user input interfaces, indirect injection through external data sources consumed by the model, and multi-turn injection techniques that establish false context across a conversation to enable later exploitation.
LLM02: Sensitive Information Disclosure. LLMs may disclose training data, system prompt contents, user data from other sessions, or information from connected data stores in response to crafted queries. GDF tests for system prompt extraction, training data memorization disclosure, cross-user data leakage in multi-tenant deployments, and retrieval of credentials or PII from connected systems.
LLM03: Supply Chain Vulnerabilities. AI applications depend on base models, fine-tuned models, embedding models, and third-party plugins that may contain security weaknesses or have been tampered with. GDF assesses the provenance and integrity of AI components in the deployment pipeline, including model weights, tokenizers, and inference infrastructure.
LLM04: Data and Model Poisoning. Training data poisoning introduces biases or vulnerabilities at the model level that persist across all uses. Fine-tuning data poisoning can insert backdoors into model behavior that are triggered by specific inputs. GDF assesses data pipeline security controls and tests for behavioral anomalies consistent with poisoning attacks.
LLM05: Improper Output Handling. When LLM outputs are passed directly to downstream systems without validation, injection attacks can propagate through the AI layer into SQL queries, shell commands, HTML rendering contexts, and API calls. GDF tests for output handling failures by analyzing how application code processes and uses model responses.
LLM06: Excessive Agency. Agentic AI systems given broad tool access and permissions present elevated risk when their actions cannot be constrained to intended scope. GDF assesses the principle of least privilege applied to AI agent tool access, tests for unauthorized action sequences achievable through prompt manipulation, and reviews human-in-the-loop controls on high-impact operations.
LLM07: System Prompt Leakage. System prompts contain confidential instructions, business logic, security guardrails, and sometimes credentials. GDF tests whether these can be extracted through direct requests, indirect techniques, or by analyzing the model's behavioral responses to probing queries designed to reveal prompt structure.
LLM08: Vector and Embedding Weaknesses. Vector databases used in RAG implementations present risks including cross-context data leakage, embedding inversion attacks, and retrieval manipulation. GDF tests vector database access controls, namespace isolation, and the accuracy of similarity search in returning only authorized content to requesting users.
LLM09: Misinformation. AI systems that generate authoritative-sounding but factually incorrect information in high-stakes contexts, such as legal, medical, financial, or technical domains, present organizational and legal risk. GDF assesses accuracy guardrails, hallucination rates on domain-specific queries, and the adequacy of human review processes for AI-generated content.
LLM10: Unbounded Consumption. AI APIs exposed without adequate rate limiting, token budgets, or abuse controls are vulnerable to resource exhaustion attacks and cost amplification. GDF tests authentication, rate limiting, and cost controls on AI API endpoints to identify conditions that could result in service disruption or significant financial impact.
GDF's AI Security Testing Methodology
AI red-team exercises at GDF are structured in four phases: scoping and architecture review, automated adversarial testing, manual red team testing, and reporting.
The scoping phase maps the AI application's architecture: model selection, system prompt structure, tool and API integrations, data sources accessed during inference, multi-model pipeline topology, and user access patterns. This mapping produces a threat model identifying the highest-priority attack surfaces before testing begins. For RAG-enabled applications, GDF documents the vector database configuration, retrieval logic, and data ingestion pipeline. For agentic systems, GDF catalogs available tools, permission scopes, and action approval workflows.
Automated adversarial testing uses GDF's library of structured attack payloads mapped to the OWASP LLM Top 10 categories. Automated testing covers high-volume prompt injection attempts across diverse injection formats, including instruction override patterns, role-play jailbreaks, encoding-based obfuscation, and context-switching techniques. This phase also includes automated extraction attempts targeting system prompt disclosure, training data memorization, and cross-user data leakage. Automated results are logged and triaged before manual testing begins.
The manual red team phase applies human judgment to develop attack scenarios specific to the client's application context. GDF's analysts design injection payloads tailored to the application's domain, user base, and connected tools. For agentic systems, analysts attempt to construct multi-step attack chains that use legitimate tool calls to achieve unauthorized outcomes. This phase includes testing of indirect injection vectors: if the application processes uploaded documents, retrieves web content, or reads email, GDF tests whether adversarially crafted external content can manipulate model behavior.
RAG security testing examines the retrieval layer specifically. GDF tests whether a user can craft queries that return content from other users' namespaces, whether embedding similarity search can be manipulated to retrieve records it should not, and whether the ingestion pipeline applies adequate sanitization to documents before they are indexed. Cross-context leakage testing verifies that the retrieval system enforces the authorization model intended by the application design.
Guardrail bypass testing assesses the robustness of content filtering, topic restriction, and output validation controls. GDF uses both direct bypass techniques and indirect methods: establishing false premises across a multi-turn conversation, using legitimate-seeming framing to approach restricted topics, and exploiting inconsistencies in how guardrails apply across different input formats or languages.
AI Red Teaming
AI red teaming is the structured simulation of adversarial attacks against an AI system by a team operating with attacker mindset and techniques. It differs from security testing checklists in that it is goal-directed: the red team defines realistic attacker objectives, develops attack strategies, and iterates against defenses, generating findings that reflect what a determined adversary would actually accomplish.
GDF's AI red team exercises are designed around specific threat actors and objectives relevant to the client's deployment context. For a customer-facing chatbot, the relevant threats include attempts to extract internal business data, generate harmful content to damage brand reputation, and manipulate the chatbot into providing unauthorized discounts or access. For an internal AI assistant with document access, the relevant threats include data exfiltration through the AI interface, prompt injection through documents uploaded by malicious insiders, and system prompt extraction that reveals security control details. For agentic systems with code execution or API access, the relevant threats include lateral movement, privilege escalation, and unauthorized data modification.
The red team exercises follow a defined kill chain: reconnaissance to understand the model's capabilities and constraints, initial access through prompt injection or guardrail bypass, exploitation to achieve the target objective, and documentation of the full attack path. Each successful attack chain is documented with the exact prompts used, the model responses at each step, and the resulting impact. This documentation supports both technical remediation and the risk assessment that security and legal teams need to evaluate the findings.
GDF also conducts adversarial robustness testing for AI models used in security-relevant classification tasks: content moderation systems, fraud detection models, identity verification systems, and malware detection classifiers. These tests apply adversarial perturbation techniques to evaluate whether the model's classification decisions can be manipulated by an adversary who controls the input.