Skip to content
Adrythm
Book 20 minutes
AI and automation

Prompt injection

prompt injection / indirect prompt injection / jailbreaking an AI / tricking a chatbot

In short

Prompt injection is when text an AI system reads becomes an instruction it follows. OWASP notes the text does not have to be visible to a person, only parsed by the model, and that no fool-proof prevention is known. The UK's national cyber security body says the same.

OWASP splits it into two cases, and the second is the one nobody pictures. A direct injection is a user's own input changing how the model behaves. An indirect injection happens when the model takes in an external source such as a website or a file, and that content changes its behavior. So the instruction can arrive inside a document you asked the assistant to summarize. OWASP adds that injected text need not be human-visible or readable, as long as the model parses it.

The UK's National Cyber Security Centre describes the same weakness. An input is designed to make the model behave in an unintended way. That can mean offensive content, confidential information revealed, or consequences in any system that accepts unchecked output from the model. Its worked example is MathGPT. A researcher saw that the model ran user-submitted text as code, used that to reach the machine hosting it, and took a sensitive API key.

Neither body claims this can be fixed. Chance sits at the heart of how models work, and OWASP says that makes it unclear whether fool-proof prevention exists. The NCSC says there are at present no failsafe security measures that will remove the risk, and advises care before putting a model into a high-risk system.

Where they agree is on what to do instead, and they arrive at it from different directions. OWASP says to give the application its own API tokens and keep those functions in code, not in the model. Cut its privileges to the minimum. Require human approval for high-risk actions. The NCSC suggests a rules-based system layered on top of the model so it cannot take damaging actions even when prompted to. Both answers constrain what the model is allowed to do rather than what it can be told.

In practice

OWASP's first example scenario is a customer support chatbot instructed to ignore its guidelines, query private data and send emails. That is the shape of most business deployments. When something is being sold to you as an AI receptionist or site assistant, the useful question is not how good its filters are, but what it is permitted to do without a person in the loop.

Not the same as

Jailbreaking
OWASP treats it as a form of prompt injection, where the input makes the model disregard its safety protocols entirely.
Data poisoning
The NCSC covers that separately: tampering with training data, rather than with what the model reads while running.

Why it matters to you

The pitch for a business AI is always that it can do things: book, look up, reply, send. Each of those is a permission, and permissions are where an injected instruction turns into damage rather than an odd answer. Two independent security bodies, one a standards project and one a government agency, both say the instruction filter cannot be relied on. That puts the safety of what you buy in its permissions, not its wording.

What to ask or check

  1. 01What can this assistant do on its own, and what waits for a person to approve?
  2. 02Does it read outside content such as webpages, uploaded files or forwarded email?
  3. 03When it connects to another system, does it use its own limited credentials or yours?

What people get wrong

That it takes someone typing something clever into a chat box. OWASP's indirect case is the model accepting input from external sources such as websites or files, so the instruction can arrive inside content the assistant was simply asked to read.

Red flags

  • A vendor who says their prompt is secure, or that injection is a solved problem.
  • An assistant that can send mail or change records with no approval step.
  • No answer to what happens when the content it reads contains instructions.

Who owns it

Whoever set the assistant's permissions. Both OWASP and the NCSC put the remedy in the surrounding system rather than in the model, which means it is a build decision and can be asked about before you sign.

Where you will see it

Any AI assistant on your site or phone line that reads customer messages, documents or webpages.

Keep reading

Privacy-enhanced AI

Privacy-enhanced AI means an AI system built so your data is not exposed or quietly reused. NIST lists it as one of seven characteristics of trustworthy AI, and states the honest catch: privacy techniques can cost accuracy. The FTC has required firms to delete models built on unlawfully obtained data.

AI voice calls

An AI voice call uses a synthesized or cloned voice to speak to the person who answers. In February 2024 the FCC confirmed those voices fall under the existing rules on artificial or prerecorded voice, so the same consent requirements apply. The message must also identify your business and offer an automated way to opt out.

Training data

Training data is what a model learned from. The question owners ask is whether their own data joins it, and there is no single answer: OWASP names three separate stages, pre-training, fine-tuning and embedding. A commitment worth having names the stage and comes in writing.

Retrieval augmented generation

Retrieval augmented generation is how an AI answers from your documents without being trained on them. OWASP describes it as combining a pre-trained model with external knowledge sources at answer time. So the documents sit in a store the system reads from, and who can read that store is the question.

AI agent

An AI agent is a model that has been granted the ability to take actions, not just produce text. OWASP says the damage one can do comes from three grants: excessive functionality, excessive permissions and excessive autonomy. What it is allowed to do matters more than how good it is.

Hallucination

A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes