Map the application and its trust boundaries

Start with an architecture and data-flow review. Identify the model and provider, system and developer instructions, user interfaces, orchestration code, memory, logs, vector stores, external retrieval sources, tools, downstream systems, and human approvals. Record which components are trusted, which are user-controlled, and which can be modified by a third party. Include development, test, and production paths because a weak non-production integration can expose credentials or become a route into a higher-value system.

The assessment should establish what the application is intended to do and what it must never do. That baseline makes later findings concrete. For example, a support assistant may be allowed to summarize a customer record but not disclose another customer’s record, change an account without verification, or reveal hidden instructions.

Test prompt injection and instruction handling

Prompt injection is an application security concern because an attacker may influence instructions through direct user input or indirect content. Test both. Direct cases attempt to override policies, extract hidden prompts, change roles, or induce unsafe actions. Indirect cases place instructions in web pages, documents, tickets, emails, or retrieval records that the application will later process. The test should examine how the system separates data from instructions and whether the application validates output before using it.

Do not limit testing to a collection of clever strings. Vary language, encoding, message order, document placement, and multi-turn context. Observe whether a successful injection can cross a meaningful boundary, such as accessing data, invoking a tool, influencing a decision, or poisoning memory. Findings should distinguish model behavior from application controls so that remediation targets the right layer.

Check data exposure and retrieval provenance

Assess what information can enter prompts, context, logs, traces, evaluation datasets, and provider systems. Test tenant isolation, authorization filters, sensitive-data handling, retention, redaction, and error messages. Attempt to retrieve records through normal questions, crafted questions, conversation history, and combinations of metadata that may reveal information indirectly. Verify that access decisions are made by a trusted application layer rather than inferred from the model’s answer.

Retrieval-augmented generation adds a provenance problem. The system should be able to identify where context came from, whether it is current, what permissions applied, and whether it was altered. Test malicious or stale documents, conflicting sources, poisoned embeddings, over-broad retrieval, and citations that do not support the answer. Provenance does not make content trustworthy by itself, but it gives people and controls information needed to evaluate it.

Evaluate tool authorization and action safety

Tools turn language into capability. Inventory every function the model can call, including hidden or indirect tools, and document the identity and privileges used for each call. Test whether a user can cause unauthorized reads, writes, messages, purchases, code execution, or changes to security settings. Check argument validation, allowlists, rate limits, approval gates, transaction boundaries, replay resistance, and logging. A tool should not receive broader authority merely because the model can phrase a plausible request.

Authorization must account for the actual user, the requested resource, the action, and the business context. A model-generated explanation is not proof of authorization. High-impact actions should have controls that remain effective when the model is wrong, manipulated, or unavailable. Where possible, design operations to be reversible and require confirmation for consequences that a user may not anticipate.

Probe business logic and operational failure

LLM applications can fail in ways that conventional prompt tests do not reveal. Test workflow rules, approval sequencing, pricing or eligibility logic, duplicate requests, escalation paths, and the difference between a recommendation and an executed action. Explore ambiguous inputs, partial outages, model refusals, hallucinated identifiers, long conversations, and conflicting instructions. Confirm that downstream systems reject invalid states instead of trusting the assistant to preserve them.

The deliverable should connect each issue to a concrete boundary and owner. Include reproduction steps, affected data or action, preconditions, evidence, severity rationale, recommended control, and retest criteria. Repeat the assessment after material changes to the model, prompts, retrieval corpus, tools, or authorization model. An assessment is evidence about tested conditions, not a guarantee that the application will resist every future attack.

Further reading

This article provides general security guidance. Assessment scope and controls should be tailored to your systems, obligations, and operating context.