Nobody can promise that an AI assistant will never be tricked. That is exactly why AI penetration testing exists. It measures how far a trick can go: what a manipulated model could read, send or change in your business, and which controls stop it.
This article covers what a test involves, the main risks it looks at, and how to scope one sensibly.
Table of Contents
What is different about testing AI systems?
A normal application follows fixed rules. Given the same input, it does the same thing. A large language model (LLM) does not work that way. It reads text, predicts a sensible reply and can be steered by the words it is given.
That creates a problem, because developers often put their own instructions and untrusted content into one prompt. The NCSC, the UK’s national cyber authority, puts it bluntly: LLMs do not enforce a security boundary between instructions and data. So a line hidden in an email, a web page or a CV can be read as a command.
Standard web application testing still matters here, because the app around the model has the usual flaws. However, it will not tell you whether the model can be talked into misusing its access. That needs its own tests.
What does AI penetration testing cover?
Most AI penetration testing combines ordinary application checks with attacks aimed at the model. A good scope usually covers these areas:
- Direct prompt injection. The tester types instructions that try to override the system’s rules.
- Indirect prompt injection. Hostile instructions are planted in content the AI later reads, such as documents, tickets, web pages or emails.
- Data exposure. Can the model be pushed to reveal customer data, other users’ chats or its own hidden system prompt?
- Tool and API abuse. If the AI can send email, query a database or call an API, can an attacker make it do so on their behalf?
- Output handling. Does the app trust what the model says and pass it straight into a browser, a database query or a shell?
- Access control. Does the AI respect who the user is, or can one customer see another’s records through it?
- Resource abuse. Can someone run up large bills or knock the service over with expensive requests?
Which risks does the OWASP LLM Top 10 list?
Testers usually map their work to the OWASP Top 10 for LLM Applications. The 2025 edition names ten risks:
- Prompt injection
- Sensitive information disclosure
- Supply chain weaknesses
- Data and model poisoning
- Improper output handling
- Excessive agency
- System prompt leakage
- Vector and embedding weaknesses
- Misinformation
- Unbounded consumption
Two of these deserve special attention for most small and mid-sized firms. Excessive agency means the AI has been given more power than it needs. Improper output handling means the app believes the model without checking. Both turn a clever trick into real damage.
Can prompt injection be fixed completely?
Not today, so that shapes how you should read any test result. The NCSC says there is “a good chance prompt injection will never be properly mitigated” in the way SQL injection was. SQL injection has a clean cure in parameterised queries. LLMs have no equivalent.
Instead, the NCSC’s advice is to reduce the impact. In practice, that means giving the AI the least access it needs, using ordinary non-AI controls to limit what it can do, and logging its inputs, outputs and tool calls. It also warns buyers to be wary of anyone who claims they can “stop” prompt injection.
So a test will rarely end with a clean pass. Instead, it shows you what an attacker could reach if the model were fooled, and which fixes cut that down.
How does an AI penetration test work?
Because the process mirrors other engagements, the steps look familiar to anyone who has bought a penetration test before.
1. Scoping
First, you agree what is in scope: the chatbot, the model provider’s API, the data sources behind it and the tools it can call. You also agree which test accounts and data to use.
2. Mapping the system
Next, the tester works out what the AI can see and do. Which documents does it read? Which actions can it trigger? Where does its output go next? This map decides where the risk sits.
3. Attacking it
Then come manual attacks, usually with some automation for volume. The tester tries injection, data extraction, role confusion, abuse of tools and the usual web flaws in the surrounding app. Because model replies vary, each attack is repeated many times.
4. Reporting and retesting
Finally, you get findings ranked by real impact, with steps to fix each one. After you make changes, a retest confirms they hold.
Who needs AI penetration testing?
You should consider it if any of the following is true:
- A customer-facing chatbot can see account or order data.
- An internal assistant searches company files, email or chat.
- An AI agent can send messages, change records or call other services.
- You build software on a third-party model and sell it to clients.
- A customer, insurer or regulator has asked how you secure your AI use.
If you only use a public chatbot with no company data, a test is less urgent. A clear staff policy on what to paste into it may be enough.
What should you ask a provider before booking?
Quality varies, and the field is young. So these questions help when you compare quotes for AI penetration testing:
- Will humans do the testing, or is it an automated scan with a new label?
- Do they test the whole app and its integrations, or only the model?
- Which framework do they map to, and can they show a sample report?
- Do they promise to “stop” prompt injection? If so, treat that as a red flag.
- How do they handle repeat runs, given that model answers change?
How can you reduce AI risk before a test?
Some steps cost little and help straight away:
- Give the AI only the data and tools its job needs.
- Require a human to approve any action that sends, deletes or pays.
- Treat anything the model outputs as untrusted input to the next system.
- Keep logs of prompts, replies and failed tool calls, and review them.
- Do not store secrets or keys in system prompts.
If you want a scoped test of an AI feature, the team at Aardwolf Security can help through its penetration testing services. You can also get in touch to talk through what you have built before you decide.
Frequently asked questions
Is AI penetration testing the same as AI red teaming?
The terms overlap, but they differ in focus. Red teaming often means a broader, goal-led attack, while a penetration test follows an agreed scope and ends with a fix list. Many providers use the two names loosely, so check what is actually being done.
How long does an AI penetration test take?
It depends on how many features, data sources and tools are in scope. A single chatbot with few integrations is a smaller job than an agent wired into several systems. A provider should quote after scoping, not before.
Does the model vendor already secure this for me?
Partly. The vendor secures its own model and platform. You still own your prompts, your data connections, your permissions and the app around it. Those are where most of your risk sits.
Will a test break my chatbot?
It should not, because testers usually work against a staging copy or with agreed limits, and they flag anything that could cause harm before they try it.
How often should we test?
Run AI penetration testing before launch and after major changes, such as a new model, a new data source or a new tool. Those changes alter what an attacker can reach.
Subscribe to our newsletter
Honest updates, straight to your inbox. Unsubscribe any time.
