Skip to content

Comparison

Penetration test or security review: which does your software need?

By Rare Agentics engineering teamLast updated First published 7 minute read

A penetration test is an authorised attack simulation by specialist testers, ending in a report of what they could break into; a security review is an engineering-led look at how the software is built, ending in fixes. Choose a certified penetration test when a customer, insurer or regulator asks for one by name. Choose a security review when you need the weaknesses found and fixed, which for most early products, and most apps built with AI tools, comes first.

Why do people confuse the two?

Both a penetration test and a security review look for weaknesses in your software, both end in a written report, and both are often called a security test. Suppliers use the words loosely too.

The difference is in the method, the output and who asks for it. Getting it wrong costs money: buying a penetration test when you needed fixes leaves you with a list of problems and nobody to solve them, and buying a review when a customer contract demands a certified test leaves the contract unsigned.

We run security reviews, not penetration tests, so we have an interest here. When a customer or regulator needs a test, we say so and work alongside the firm you choose.

What is a penetration test?

A penetration test is an authorised attack simulation. Testers try to break into your application, network or infrastructure the way an attacker would, within a scope and a time box agreed in advance.

Tests are often described by how much the testers are told. In a black-box test they start with almost nothing, like an outside attacker. In a grey-box test they get a user account and some documentation. In a white-box test they see the code and the architecture.

The output is a report: each weakness found, how it was exploited, a severity rating and a recommendation. Fixing is usually left to you, and a retest may be included for a short window or charged separately, so ask.

In the UK, buyers sometimes ask for testers accredited under a recognised scheme. CREST is the industry body most often named in commercial contracts. CHECK is the National Cyber Security Centre scheme for testing government, public sector and critical national infrastructure systems. CREST accredits firms and certifies individual testers, so ask which one a contract means. If a contract names either, you need a firm that holds that accreditation.

Cyber Essentials is not a penetration test either. It is a government-backed certification of basic controls, and Cyber Essentials Plus adds a hands-on technical check of those controls. A customer asking for Cyber Essentials is asking for something different from either a test or a review.

What is an engineering-led security review?

A security review looks at how the software is built, from the inside, by engineers who read the code, the database rules, the cloud configuration and the flows people use.

It covers many of the same weaknesses a tester would hunt for, but it finds some that an outside test is unlikely to: an access rule that is too broad but hard to reach, a key that sits in the repository history, a backup nobody has ever restored, or customer data sent to an AI provider without anyone checking what it keeps.

The output is fixes, not just findings. In our review every finding is reproduced and ranked, we fix what we find within the agreed price or guide your team, every fix gets a test, and a second pass confirms it.

How do they compare side by side?

Penetration testEngineering-led security review
The question it answersCould an attacker get in, and how far?Where is this software weak, and how do we fix it?
MethodAttack simulation, usually from outsideReading code, rules and configuration, and probing it
Who does itSpecialist testersEngineers who build software
Access neededOften little or noneRead access to the code and the accounts in scope
OutputA report of findings and recommendationsFindings, fixes, a test for each fix and a retest
FixingUsually yoursIncluded within the agreed price, or guided
AccreditationSometimes required: CREST for commercial contracts, CHECK for government workNot an accredited test
Best forContract, insurer or regulator requirementsFinding and fixing weaknesses before launch, a raise or a sale

What should either one look at?

Whichever you buy, the scope should cover the places where real incidents start. A short list to hold any supplier to:

  • Sign-in and account recovery. Password reset, magic links, invitations and email change flows, which can be abused to take over an account.
  • Access rules. Whether one signed-in user can read or change another user's records, through the screens and directly through the API.
  • Keys and secrets. Whether any secret key reaches the browser, sits in the repository or its history, or is shared between staging and production.
  • Limits. Rate limits and lockouts on sign-in, forms and any AI feature that costs money each time it runs.
  • Cloud accounts. Roles and service accounts with more access than they need, and features left on by default.
  • Outside services and AI providers. What data leaves your systems, to whom, and what they keep.
  • Backups. Whether a restore has ever been proven, not just whether backups exist.
  • People. Access that outlives the person who needed it, such as a former contractor's admin account.

A penetration test will usually reach sign-in, access rules, limits and any key exposed in the browser. It will not see keys in the repository history, cloud roles, backups or former staff access. A review covers all eight, because it can see the accounts and the code.

What should you ask a supplier before you buy?

  1. What exactly is in scope, and what is out? Get it signed before any testing starts.
  2. Who will do the work, and what have they built or tested before?
  3. Is fixing included, or only finding? If only finding, who fixes?
  4. Will each finding be reproduced, so you know it is real and not a scanner's guess?
  5. Is a retest included, and does each fix get a test so it cannot quietly return?
  6. What will the report look like, and can you share it with an investor or customer?
  7. For a penetration test: which accreditation does the firm hold, and does it match what your customer or contract names?
  8. What access do they need, and when is it removed?

When do you need a certified penetration test?

You need one when somebody else requires it by name. The common cases:

  • An enterprise customer's contract or security questionnaire asks for an annual penetration test by an accredited firm.
  • Your cyber insurer asks for one as a condition of cover.
  • You handle card payments in a way that brings penetration testing into your PCI DSS scope.
  • You supply systems to government or the public sector, where a CHECK test may be required.
  • A regulator or certification audit asks for independent testing evidence.

In each case, the requirement is about evidence for a third party. Check exactly what is asked for, the scope, the accreditation and how recent the test must be, before you buy.

When is a security review the better first step?

For most early products, a review comes first, for three reasons.

You need fixes, not just a list. An early product, especially one built quickly with AI tools such as Lovable, Bolt, Cursor, Replit or v0, usually has gaps in access rules, keys and limits. A report that lists them without fixing them leaves the work undone.

An outside test is a poor use of money on obvious gaps. If the access rules are missing, a tester will find that quickly and spend the rest of the time box on it. Fix the basics first and a later test goes deeper.

Due diligence asks how it is built. Investors and buyers often want to know how the software is built and whether known weaknesses have been fixed, with evidence. A written review with fixes and retests answers that directly.

Can you use both?

Yes, and the order matters. A sensible sequence for a growing product:

  1. A security review before launch, a raise or a sale, with fixes and retests.
  2. A certified penetration test once a customer, insurer or regulator asks for one, run against software whose obvious gaps are already closed.
  3. Fixes for anything the test finds, each with a test so it does not come back.
  4. A review of each major new feature as it ships, and a repeat test on whatever cycle the contract requires.

When a test is needed, we work alongside the firm you choose: preparing the scope, fixing what they find and confirming each fix.

How do you decide which one to buy?

  • Has a customer, insurer, regulator or contract asked for a penetration test by name? If yes, buy one from a firm with the accreditation they name.
  • Is the product early, or built quickly with AI tools? Start with a review.
  • Do you have engineers who can fix what a test finds? If not, a review that includes fixes is the safer buy.
  • Is due diligence coming? A written review with fixes and retests answers most of the technical security questions investors ask.
  • Are the basics closed: access rules, keys server-side, rate limits, account recovery, backups restored at least once? If not, fix them before paying for a test.

How Rare Agentics can help

Our security and hardening review is engineering-led and quoted to scope. It takes 1 to 3 weeks, with a signed scope before testing starts, every finding reproduced and ranked, the fixes, a test for each, and a retest. For apps built with AI tools, see our security review for AI-built apps. If you also need a certified penetration test, we will say so and work alongside the firm you choose. Tell us about your software, and within two working days you get a first view.

Related reading

Related services

Start a conversation

Built it fast with AI tools? Send us the app.

Tell us what it does and what it was built with. We will tell you what we would check first, before a launch, a raise or a sale. Within two working days you get a first view, and within a week a written plan, a fixed price and the risks.

See what a security review covers

In plain language: what you want to exist, who it is for, and anything that worries you. It does not need to be technical.

Read by a senior engineer. Used only to reply to you.

Tell us what you need built.