Your AI Pen Test Might Just Be an If-Then Statement
Every FRSecure client and prospect is getting the same pitch: an AI pen test that beats a human tester, for half the price. On Episode 6 of the Truth in Cyber podcast, Oscar Minks and Michael Kennedy asked Seth Bowling, R&D manager at FRSecure, to check that claim.
Seth’s team has spent most of this year testing about a dozen automated and “AI-enabled” pen testing tools. They ranged from free open source tools to expensive commercial products, and the team compared every one against real pen tests done by people. The tools came up well short.
Where’s the AI?
The biggest surprise: most of these tools barely use AI to run the test. Under the hood, it’s old-school automation, basically a long chain of if-then rules. The AI gets added at the end to write up the findings. That’s backwards, because judging what a finding means for your business is the one thing people still do better than AI.
AI is great at reading code and spotting flaws. It’s much weaker at the moment-to-moment judgment calls a real attack takes. The cutting-edge AI models that can do more are far too expensive to run for the $3,000 to $4,000 these vendors charge for a pen test.
“If you’re doing one pen test a year, your only pen test for the year should not be through an automated or AI product, period.”
Seth Bowling, FRSecure
Spend smarter
On a tight budget? Seth’s advice is to buy a good vulnerability scanner before an automated pen test. Many of these tools rely on free open source scanners that can’t match the established commercial ones.
Automated tools still have a place. Run them weekly to keep checking your defenses, then bring in human testers a couple of times a year.
Ask before you buy
Shopping anyway? Have the vendor run a demo on your own systems, and compare the results to your last pen test. Then ask two questions: what is the AI actually doing, and where does your data go during and after the test?
Seth expects truly AI-driven testing to improve fast. For now, it’s a supplement, not a replacement.
Catch the full episode to hear the rest.
