Skip to content

LLM penetration testing, app included

LLM penetration testing attacks your chat, RAG and AI features the way an adversary would, together with the web app and API around them. For product teams who want findings their engineers can replay and fix, mapped to the OWASP Top 10 for LLM Applications 2025.

What is LLM penetration testing?

LLM penetration testing is a security test of an application built on a large language model. It attacks the prompts, the retrieval layer, the output handling and the permissions around the model to find what a real attacker could steal or trigger. Grapefruit proves each finding with the exact request and response.

It is a pentest of your product, not an evaluation of a model. The target is the chat widget, the RAG search, the summariser, the copilot inside your dashboard. The model is one component. The test follows an attack wherever it goes.

Grapefruit is the pentest for products that ship AI. The same run that attacks your login, your sessions and your API attacks the LLM feature sitting on top of them, and everything lands in one portal.

How is LLM penetration testing different from AI red teaming?

AI red teaming usually probes the model itself: can it be pushed into harmful, biased or off-policy output. LLM penetration testing probes your application: can an attacker use the model to reach data, accounts and backend functions. Red teaming asks what the model will say. A pentest asks what your product will do.

If you build on someone else's model, its raw behaviour is mostly out of your hands. Your prompts, retrieval, tools and APIs are entirely in them. That is where an LLM pentest spends its time, and where the fixes are yours to ship.

Grapefruit includes red-team style attacks, such as jailbreaks and prompt extraction, as steps in a chain and keeps going until the impact on your product is clear.

Why test the LLM feature with the app and API around it?

Because the damage happens where the model's output meets your backend. A manipulated answer is a nuisance. A manipulated answer that becomes a database query, a rendered script or an API call with someone else's ID is a breach. Grapefruit tests web app, API and LLM feature in one engagement so those chains surface.

Most vendors sell AI red teaming and app pentesting as two projects with two reports. Attackers do not respect that split. A typical chain starts in the chat box, passes through a retrieval step and ends at a REST or GraphQL endpoint that trusted the model.

  • Chat to database: the model builds a query from user text, and the query runs with more access than the user has.
  • Chat to browser: the model returns markup, your front end renders it, and script runs in another user's session.
  • Document to data leak: a poisoned file in the RAG store instructs the model to include another customer's records in its answer.
  • Prompt to API: a leaked system prompt names an internal endpoint, and that endpoint skips an authorization check.

Which OWASP LLM risks does the test cover?

Coverage maps to the OWASP Top 10 for LLM Applications 2025, from LLM01 Prompt Injection to LLM10 Unbounded Consumption. The web and API side follows the OWASP Web Security Testing Guide (WSTG) v4.2 and the OWASP API Security Top 10 2023, so one report speaks to all three.

The risks weigh differently in every product. A RAG search over shared documents lives or dies on LLM02 and LLM08. A copilot that can write to your database is an LLM05 and LLM06 story. The test goes where your feature actually has reach.

IDRiskWhat we testWhat a finding looks like
LLM01Prompt InjectionDirect and indirect instructions through chat, files, retrieved content and tool resultsThe model follows the attacker's instruction over yours
LLM02Sensitive Information DisclosureAnswers, retrieval results and error messagesAnother user's data, secrets or internal details returned in a response
LLM03Supply ChainThird-party plugins, tools and integrations your LLM feature exposesA dependency the feature trusts becomes the way in
LLM04Data and Model PoisoningContent users can write into knowledge bases and feedback loopsPlanted content changes the answers served to other users
LLM05Improper Output HandlingHow model output is rendered, stored and passed to other systemsModel output executes as script, query or command
LLM06Excessive AgencyThe tools and permissions the model can useThe model performs an action the user was never allowed to take
LLM07System Prompt LeakageHidden instructions and anything embedded in themThe full prompt recovered, with rules, endpoints or keys inside
LLM08Vector and Embedding WeaknessesRAG retrieval and tenant separation in the vector storeDocuments from one tenant retrieved for another
LLM09MisinformationAnswers users act on in sensitive flowsAn attacker steers the feature into stating false policy or account details as fact
LLM10Unbounded ConsumptionRate limits, token limits and cost controls on LLM endpointsOne account drives runaway usage and cost

What do you get from an LLM pentest?

You get findings ranked critical, high, medium or low in an online portal, each with the captured request and response that proves it, plus a downloadable report. Your engineer pastes the request, runs it and watches the attack work. After the fix, the same request fails, and you have your confirmation.

Proof changes what gets fixed. Cobalt's 2025 State of Pentesting data, as reported by CSO Online, shows 48% of pentest findings get fixed, 69% of serious ones, and only 21% of findings in generative AI apps. A finding nobody can reproduce sits in the backlog. A finding your engineer can run gets a ticket and an owner.

  • Severity: critical, high, medium or low
  • The exact request that triggers the issue
  • The captured response that proves it
  • A re-test after your fix
  • A downloadable report for customers and auditors

How does the test run?

The test is autonomous and uses real exploitation, so it starts when you are ready, not when a calendar opens up. You add your hostname, publish one DNS TXT record and sign the testing authorization online. That takes a few minutes, and results land within hours of kickoff.

Supply test accounts and the test runs authenticated, as a logged-in user, which is where LLM features usually live. Compare that with the traditional route: according to Stingrai, engagements commonly take 2 to 4 weeks from accepted quote to start, 5 to 10 business days of testing, and the report 1 to 2 weeks after that.

Fix a finding and it is re-tested, on the next scheduled run if you subscribe. The portal shows what got fixed and what is new.

Should you run one LLM pentest or test continuously?

Both are on offer. A full penetration test at a fixed quote gives you a deep point-in-time result for a launch, a customer request or an audit. A maintenance plan re-tests your verified domains every week or month, which suits LLM features because prompts, models and retrieval data change far more often than once a year.

Each run is dated, kept in the portal, and shows what is new and what got fixed since the last one. That record doubles as an evidence trail for a SOC 2 Type II audit window. Maintenance plans have public prices and card checkout. See pricing and start today.

Questions

What is the difference between LLM penetration testing and a regular web application pentest?
A regular web pentest attacks inputs, sessions, access control and server behaviour. An LLM pentest adds the attack surface the model creates: prompts, retrieval, tool use and model output flowing into your code. Grapefruit runs both in the same test, because an attack that starts on the LLM side often finishes in the app or API.
Is LLM penetration testing the same as AI red teaming?
They overlap. AI red teaming usually centres on model behaviour: harmful output, bias, policy violations. LLM penetration testing centres on your application and what an attacker can reach through it. Grapefruit uses red-team techniques such as jailbreaks and prompt extraction, then follows them into your data, tools and APIs to show concrete impact.
Do you need access to our model, weights or source code?
No. The test targets your deployed application over the network, the same way an attacker reaches it. You verify the domain with one DNS TXT record, sign the authorization online and supply test accounts for authenticated coverage. It works the same whether you call a hosted model through an API or run your own.
Which LLM features can you test?
Chat assistants, RAG and knowledge search, summarisers, copilots embedded in a web app, and AI agents with tools, along with the web application and REST or GraphQL APIs they sit on. If the feature is reachable on a domain you verify, it is in the test, and its findings land in the same portal as the rest.
Does an LLM pentest count for SOC 2?
SOC 2 does not formally require a penetration test, but auditors commonly expect one, and for Type II the testing has to be evidenced inside the audit period, according to Baker Tilly. Grapefruit covers your app, API and LLM features in one report, and weekly or monthly runs give you dated evidence across the whole window.
How much does LLM penetration testing cost?
Grapefruit maintenance plans have public prices on the pricing page, and a full penetration test comes at a fixed quote. For comparison, traditional web application pentests commonly cost $5,000 to $30,000, more for complex multi-role apps, according to Astra's penetration testing cost guide. See pricing for current plans.
How do we confirm a fix worked?
Replay the request from the finding. If the fix holds, the attack that worked before now fails. Subscribers also get the finding re-tested on the next scheduled run, and the portal shows what got fixed since the last run with its date, so you have a record as well as a result.

Standards and sources