Prompt Injection

Direct and indirect injection

Direct injection is a user typing something intended to override the system's instructions. It matters for products exposed to the public, where the user is a potential adversary. In an enterprise setting, where the user is an employee who already has legitimate access, it is usually the less interesting half — someone with the access does not need to trick a model to use it.

Indirect injection is the enterprise problem. The malicious instruction lives in content the AI reads on the user's behalf: a fetched page, a dependency's documentation, a ticket description, a code comment, a file in a cloned repository. Nobody in the organization wrote it, nobody reviewed it, and the user may never see it.

OWASP ranks prompt injection first in its Top 10 for Large Language Model Applications, and the indirect variant is the reason it holds that position — the attack surface is every piece of external content an AI system consumes.

Why filtering does not solve it

The obvious response is to detect and strip injected instructions. This is worth doing and it is not a solution, for a reason that is structural rather than a matter of effort.

Instructions and data share one channel. There is no delimiter a model reliably honours, and there is no vocabulary that distinguishes 'ignore previous instructions' from a legitimate sentence discussing that phrase. Pattern matching catches known phrasings; paraphrase, translation, encoding and indirection defeat it. Treat detection as raising the cost of an attack, not as closing the hole.

This is why the guidance from every serious source converges on limiting consequences rather than perfecting detection.

The defences that hold

Rank defences by whether they still work after the model has been successfully influenced. The ones that do are the ones worth building the architecture around.

• Least privilege for tools: an agent that cannot read secrets cannot be induced to disclose them.
• Human confirmation on consequential actions, especially anything that sends data outward or changes state irreversibly.
• Separating retrieval from action, so content fetched from outside cannot directly trigger a tool call.
• Action governance evaluated on the action and its context, independent of the prompt that produced it.
• Treating model output as untrusted input to whatever consumes it next — the same discipline as any other untrusted source.
• Recording attempts, so a successful injection leaves a trace rather than nothing.

Why this is a coding-agent problem specifically

A chat assistant influenced by injected text produces misleading output, which is a correctness problem. An agent influenced by the same text takes actions, which is a security problem.

Coding agents also consume unusually large amounts of untrusted content as a matter of routine: repositories they did not write, issues filed by outsiders, packages pulled from public registries, documentation fetched from the web. The exposure is continuous rather than occasional.

What Oconee Runtime does here

Detection covers unsafe instruction patterns and content from untrusted sources, contributing to the risk signal attached to an action. That is the upper layer of the diagram above, and it is treated as a signal rather than as a guarantee.

The weight sits on the layer below it: actions are evaluated on what is being attempted and in what context, so an action triggered by injected content is judged the same way as one triggered by a person. That property — indifference to how the instruction arrived — is the reason action governance is the durable defence.

Examples

• A GitHub issue contains hidden text instructing any AI agent reading it to add a new dependency from an unfamiliar registry.

  Detection may flag the instruction pattern. What actually stops it is that a dependency change is a governed action class requiring confirmation, whatever prompted it.

• A fetched web page instructs the assistant to summarize the user's environment variables into its reply.

  Least privilege is the primary defence: if the tool set cannot read the environment, the instruction has nowhere to go. Where it can, reading credentials is a governed action and produces a decision point and a record.

• A dependency's README asks the agent to push a branch to an external remote.

  Destination control makes the remote the deciding factor. A push to an unexpected remote is blocked on the operation, without any need to determine that the README was hostile.

Frequently asked questions

Can prompt injection be fully prevented?

Not by input filtering, and no published approach claims otherwise. It can be made substantially less consequential by limiting model capabilities, inserting human decision points before consequential actions, and governing actions independently of the prompt.

Is prompt injection the same as jailbreaking?

They overlap but differ in target. Jailbreaking aims to make a model produce content its provider restricts. Prompt injection aims to make a model act against its operator's intent — and in an agent, acting is the point.

Does using an enterprise or private model deployment fix it?

No. A private deployment addresses where data goes and who else can see it. Injection is about what the model does with content it reads, and that is unchanged by where it runs.

How would we know an injection succeeded?

Only from evidence. The signature is an action inconsistent with the user's stated intent — an unexpected network call, an unrelated file write, a push to somewhere new. Without a record of attempted actions and their context, a successful injection leaves nothing behind to find.

Sources

• OWASP Top 10 for Large Language Model ApplicationsOWASP
• OWASP GenAI Security ProjectOWASP
• MITRE ATLAS — adversarial threat landscape for AI systemsMITRE
• Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1)NIST