LLM06_ Sensitive Information Disclosure

OWASP's Top 10 for LLM Applications category for an AI model or the application wrapped around it revealing sensitive information it shouldn't - training data memorization surfacing in generated output, a system prompt containing secrets that can be extracted through careful querying, or a RAG (retrieval-augmented generation) pipeline surfacing documents the querying user shouldn't have access to. This is specific to applications built around large language models, a narrower and newer surface than traditional web/API vulnerability classes.

Probing for this requires understanding the specific LLM application's prompt structure, training/fine-tuning data, and retrieval permissions model, then crafting adversarial prompts designed to extract information across that specific boundary - a fundamentally different testing methodology from HTTP-level vulnerability scanning, closer to red-teaming a model's behavior than probing a web endpoint.

LLM-specific red-teaming and prompt-injection testing frameworks, and manual review of what is included in system prompts, training data, and retrieval permission boundaries for RAG-based applications.

Tip: Adversarial red-teaming, deliberately crafting prompts and indirect content designed to override intended behavior, is the standard discovery method here, since there's no static payload signature the way there is for a classic injection bug; this needs to be an ongoing exercise given how quickly new bypass techniques emerge.

Real-World Impact

Vulnerabilities specific to LLM-integrated applications are a genuinely new category rather than a repackaging of classic web bugs: prompt injection lets an attacker's text (in a document, a webpage, a user message) redirect the model's behavior away from its intended instructions, sometimes well enough to exfiltrate data or trigger actions the application exposes to the model. Sensitive information disclosure through a model can happen even without a classic data-access bug, if the model was trained on, given access to, or can be manipulated into revealing data it shouldn't surface. Model denial-of-service exploits the disproportionate compute cost of certain inputs to degrade or exhaust a service cheaply. Supply-chain risk extends the familiar dependency-security problem to models, datasets, and plugins pulled from external sources.

What makes this family distinct from traditional application security is that the "input validation" boundary is fuzzy by design — the model is meant to process arbitrary natural language, which is exactly what makes filtering malicious instructions out of legitimate ones a much harder problem than validating a structured API parameter.

As LLM integration into product surfaces accelerates, this category is also one of the fastest-evolving on this site — techniques and mitigations that were considered sufficient a year ago are routinely bypassed by newer prompt-injection and jailbreak techniques, which makes ongoing red-teaming more important here than a one-time review.

Prevention and Secure Design

Preventing LLM06_ Sensitive Information Disclosure takes a defense-in-depth approach — no single control below is sufficient alone, but together they close off both the primary path and the most common bypasses.

Treat all model input as untrusted, including retrieved content. Text pulled into a prompt from a document, webpage, or tool result is just as untrusted as direct user input — apply the same skepticism to indirect prompt-injection vectors as to direct ones.

Enforce the real security boundary outside the model, not inside the prompt. Whatever actions or data access the model can trigger should be gated by conventional authorization checks in the surrounding application code — never rely on the model's own instructions as the actual security control.

Limit what the model can access and do to the minimum required. Scope tool access, data access, and action permissions narrowly per use case, so a successful prompt injection has a small blast radius rather than the full capability surface.

Rate-limit and bound compute-expensive requests. Cap input length, generation length, and request rate to limit exposure to model-denial-of-service patterns.

Vet models, datasets, and plugins with the same rigor as any other dependency. Verify provenance and monitor for unexpected changes, the same discipline applied to traditional software supply-chain security.

Red-team continuously, not once. New prompt-injection and jailbreak techniques appear regularly — an assessment from even six months ago should not be assumed to still hold.