Do Son 
A company can approve an AI assistant without fully understanding what it will be able to see. A chatbot that summarizes public documents presents a different security problem from one that searches internal customer records. Give the same assistant permission to update a CRM or issue refunds, and an inaccurate or manipulated response can become an unauthorized transaction.
These distinctions should shape an AI readiness assessment before procurement or development moves ahead. The useful starting point is a map of the data, identities, and actions involved in each proposed workflow. It reveals which generative AI security risks arise from the model and which come from the surrounding application.
What Will the AI Actually Have Access To?
Consider a proposed assistant for a customer support team. Initially, its purpose is to draft answers using published help articles. Later, the product team connects it to support tickets and account histories so responses can reflect a customer’s circumstances. Eventually, it may be asked to change subscription details or process refunds.
These are three different deployments from a security standpoint, even if employees see the same chat interface. The first uses public information. The second processes restricted records. The third can take actions with financial consequences.
Before connecting the assistant to internal systems, examine the permissions it will actually use. Support employees may have access only to their assigned customer accounts, while a shared service account could give the assistant access to the entire customer database. Existing SaaS products need similar scrutiny, especially when new AI features can use permissions previously granted to the application.
For the initial rollout, the support assistant could retrieve customer information and draft responses without permission to modify records. If the company later decides to automate subscription changes or refunds, those functions will need additional authorization checks and, for higher-risk transactions, human approval.
Where Will Customer and Company Information Travel?
In an AI application, the submitted message is only part of the request. The software may append chat history, retrieved ticket excerpts, customer identifiers, tool results, as well as instructions for the model. Support staff may see a short question while the underlying request includes several pages of sensitive information.
Trace that complete payload and every place it can persist. Copies of this information may be stored in application logs, monitoring tools, conversation databases, retrieval caches, and also the model provider’s systems. Data leakage can result from ordinary authorized use if more information is transmitted or retained than the task requires.
For the support assistant, the application could retrieve a ticket summary and the few account fields needed to answer a billing question instead of attaching an entire customer history. Removing unnecessary fields before the model call is more reliable than asking the model to avoid mentioning them. Redaction may help, although replacing names with identifiers does not necessarily make the remaining information anonymous.
Vendor review has to cover the exact service and contract the company will use. Among third-party model providers, policies for training, retention, monitoring, and regional processing may differ between consumer products, enterprise offerings, as well as APIs. A commitment not to train on submitted content does not, by itself, establish whether that content is logged or how long it remains available.
Check which subprocessors can access company data, how deletion requests are handled, and also what the contract requires the provider to disclose after a security incident. If the provider cannot meet the company’s requirements, self-hosting may be an option. Running the model on internal infrastructure gives the company greater control over data processing, although its own team must maintain the environment, apply security updates, as well as protect model files. These responsibilities can be substantial, especially for companies without the resources or expertise to manage AI infrastructure.
Whose Permissions Determine What the Assistant Can Retrieve?
Retrieval-augmented generation, or RAG, lets an application find relevant documents and place excerpts in a model’s context. It does not automatically carry over the source system’s permission rules.
When support tickets, internal escalation notes, and also account records are stored in the same search index, the assistant may retrieve information that an employee is not authorized to view. This can happen if the search service uses an account with broad permissions. As an example, a routine customer inquiry could return a confidential account note that the support agent cannot access through the company’s CRM.
In order to prevent this, the application should enforce access controls based on the employee’s permissions before passing retrieved information to the model. Some systems can restrict access at the document level. Others require more granular controls for individual records and fields. Links to source documents should also follow the permissions established in the original system. Simply instructing the model to protect confidential information cannot enforce the restrictions in question.
The company should also test how the assistant handles changes in employee permissions. Someone who moves to another department or loses access to a customer account should no longer be able to retrieve the associated records, including through cached search results. Security testing should cover different user roles, revoked permissions, and queries involving both public and restricted information. OWASP identifies unauthorized information disclosure across users or contexts as one of the security risks associated with vector and embedding systems.
What Happens When an Outside Document Contains Instructions?
Support tickets, emails, and uploaded attachments can contain malicious instructions intended to manipulate an AI assistant. For instance, an attacker could add text to a support ticket telling the assistant to retrieve another customer’s records and send them to an external email address. If the assistant follows these instructions, the attacker may gain access to confidential information. This type of attack is known as indirect prompt injection, where malicious instructions are placed in content the AI processes as part of its legitimate task.
The risk increases when an assistant can interact with other systems or perform actions through APIs. An assistant designed to summarize support tickets may also have access to tools for sending emails or updating customer records. If these tools have excessive permissions, a successful prompt injection attack could lead to unauthorized data access or changes to business systems.
In accordance with OWASP’s guidance on prompt injection, instructions alone cannot reliably prevent AI models from following malicious content. Therefore, developers should restrict the tools available to each AI function and enforce authorization checks in the application code. For example, an assistant responsible only for summarizing tickets has no reason to access email-sending functions. Where such functions are required, the application should validate recipients, destinations, and also requested actions before executing them. API credentials should also remain inaccessible to the model.
When Does an Answer Become an Action?
The boundary between advice and execution deserves separate approval. A system may be accurate enough to suggest a refund while remaining unsuitable for issuing refunds automatically.
Before the assistant can execute a transaction, the application should verify that the employee has permission to perform the requested action. It should also check the account details, transaction amount, and also relevant business rules. Operations involving large refunds or exceptions to standard procedures may require approval from an authorized employee. In such cases, the reviewer should see the exact action the assistant intends to perform, including the transaction details.
The company should also evaluate responses that don’t trigger actions. An assistant may provide outdated policy information, combine details from different procedures, or, for instance, cite documents that don’t support its answer. Testing should establish whether responses accurately reflect the available sources and how the assistant handles questions it cannot answer reliably. If the system generates structured data, the application should validate its format and contents against the required rules. AI-generated code should undergo the same security reviews and testing as developer-written code.
Tests should reflect how employees will use the assistant, including common requests, ambiguous questions, and situations where the necessary information is missing. They should also include documents containing malicious instructions to assess the system’s resistance to prompt injection. Before deployment, the company should define which types of errors are unacceptable and what test results are required for approval. These criteria should reflect the risks of the intended workflow, since a general accuracy score may conceal failures with serious business or security consequences.
Who Will Notice if Those Controls Stop Working?
Security approval cannot end when the pilot passes. A newly enabled connector, a changed vendor retention setting, or an expanded tool permission can alter the original risk assessment without changing the employee interface.
Capture enough information to investigate incidents: requester identity, documents retrieved, tool-call attempts, authorization decisions, and relevant model or application versions. Avoid indiscriminate retention of full prompts, which can create another sensitive data store. Assign an owner who can suspend a connector, revoke credentials, identify affected records, and coordinate with the provider when a suspected exposure occurs.
The company’s AI usage policy should specify approved applications, acceptable data categories, prohibited integrations, and also how employees request exceptions. It should also cover embedded AI features employees may activate within already approved software. NIST’s Generative AI Profile offers a voluntary framework for identifying as well as managing risks across an AI system’s lifecycle.
A sensible initial release may therefore be deliberately narrow: one department, a constrained document collection, read-only permissions, and a defined review date. Expansion becomes easier to justify when the team can demonstrate that authorization, monitoring, and response procedures work under realistic conditions. The goal of prelaunch review is to decide exactly which capabilities the company can safely authorize today, and which require more engineering before employees receive them.