LLM security: the risks of language models and how companies control them
LLM security covers the measures companies use to control the specific risks of language models: inputs that manipulate the model's behaviour, outputs processed onward unchecked, and permissions that reach further than necessary.
The decisive difference from classic application security: a language model has no clean separation between instruction and data. Both arrive as text, and the model cannot reliably tell which of it is a legitimate instruction.
This article sorts the risks that affect deployers – companies using models rather than building them – and names the countermeasures in order of effect.
Direct and indirect prompt injection
Prompt injection is the attempt to make a model override its instructions through text. In the direct variant the user types that text themselves. It is the smaller problem, because the person is only exercising rights they already have.
The indirect variant is the relevant one: the instruction sits in content the model reads on the user's behalf – an uploaded document, a web page, an email, a calendar entry. The model processes it as text and can take it as an instruction.
That turns every system reading external content into an attack surface. The attacker needs no access to the company – it is enough that a document with prepared content arrives there and gets read.
The risks that affect deployers
Companies using models rather than building them do not have to engage with every risk class. These six are the relevant ones; the full taxonomy is maintained by the OWASP GenAI Security Project.
| Risk | What happens | Most effective countermeasure |
|---|---|---|
| Indirect prompt injection | Prepared text in a document or on a web page steers the model | Treat external content as untrusted; derive no automatic actions from content that was read |
| Leakage of confidential data | Personal data or trade secrets reach a system with no contract and no location commitment | A central platform with a data processing agreement and a clarified region instead of private accounts |
| Unchecked downstream use of output | Model output lands unfiltered in documents, emails, code or database queries | Treat output as a draft; validate technically before it flows into systems |
| Overly broad permissions | An assistant may do more than the use case requires – writing rather than only reading | Minimise rights per use case; require confirmation for write actions |
| False statements that look like facts | Figures, deadlines or legal bases are presented convincingly but incorrectly | Mandatory source verification; prefer applications that return citable passages |
| Weaknesses in the knowledge base | Content that not every user may see ends up in a searchable store | Model access rights at document level rather than filtering only the answer |
Why hallucinations are a security topic
Incorrect model output is usually treated as a quality problem. It becomes security-relevant as soon as it feeds into decisions: an invented deadline in a pleading, a wrong legal basis in a decision, a non-existent rule in an internal instruction.
Hallucinations cannot be excluded technically. They can be contained organisationally, by treating output as a draft by default and making source verification mandatory for certain categories – figures, deadlines, legal bases, quotations.
Applications that draw on your own stored documents and return citable passages lower the risk considerably, because verification then means looking something up rather than researching it.
Permissions: the risk grows with the rights
As long as a model produces text, the damage stays contained. It grows sharply once the model may act – sending emails, writing files, changing records, calling other systems.
That combination is what makes indirect prompt injection dangerous: a prepared document can then not merely distort an answer but trigger an action. The same rule therefore applies to assistants as to service accounts – as few rights as possible, and write actions only with explicit confirmation.
Frameworks to orient by
There is no need to invent the taxonomy yourself. Three established frameworks cover different layers and combine well.
- OWASP GenAI Security Project: maintains the top ten risks for LLM applications – the technical reference for threats and countermeasures.
- NIST AI Risk Management Framework 1.0, published in January 2023: structures risk management into the four functions Govern, Map, Measure and Manage – useful as the skeleton for your own governance.
- ISO/IEC 42001, published in December 2023: the first certifiable standard for AI management systems – relevant where clients or regulators require evidence.
Measures in order of effect
The order is sorted by effect per unit of effort. The first three steps cover most of the realistic risk.
- Centralise usage: one managed access route with a contract, a clarified region and training excluded, instead of private accounts. That removes the most common loss event.
- Define output as a draft and record the verification duty for figures, deadlines and legal bases in writing.
- Minimise rights per use case; allow write actions only with confirmation.
- Treat external content as untrusted – especially where documents or web pages are read automatically.
- Model access rights in the knowledge base at document level rather than filtering only the answer.
- Log what was used, when and by whom, so that a suspicion can be checked at all.
- Train people: anyone who has seen indirect prompt injection once in an example recognises it later. That also addresses Art. 4 of the EU AI Act.
Frequently asked questions
What is prompt injection?
The attempt to make a language model override its instructions through text. Directly, the user enters that text themselves; indirectly, it sits in content the model reads – a document or a web page, for instance. The indirect variant is the more relevant one for companies.
Can prompt injection be prevented technically?
Not entirely, because language models cannot cleanly separate instruction from data – both arrive as text. The risk becomes manageable through architecture: treat external content as untrusted, minimise rights, and derive no automatic actions from content that was read.
Are hallucinations a security problem?
Once output feeds into decisions, yes. An invented deadline or legal basis works like incorrect advice. The effective countermeasure is organisational: output counts as a draft, and verification at the source is mandatory for figures, deadlines and legal bases.
How much does the choice of model matter?
Less than commonly assumed. The most frequent loss events come not from the model but from the environment: no contract, an unclear processing location, overly broad rights, unchecked downstream use. Model choice is a quality question; the environment is a security question.
Do we need ISO/IEC 42001 certification?
Only where clients, regulators or tenders require evidence. As an organising framework the standard is useful without certification. For most companies in the DACH region, a documented AI policy with training is the more pragmatic first step.
What is the most common mistake in practice?
Anchoring security to the model rather than the environment. By far the most frequent loss event is not an attack but everyday work: confidential data in a private account, with no contract, no clarified region and no record of what went there.
Sources
- OWASP Top 10 for LLM Applications · OWASP GenAI Security Project
- AI Risk Management Framework (AI RMF 1.0) · NIST
- Regulation (EU) 2024/1689 (EU AI Act) · EUR-Lex
Related reading
Bring AI into your company securely.
Try Custodos with your team – and see how quickly secure AI becomes productive.
- Try it with the whole team
- Set up in minutes
- Productive from day one