I Ran a Pentest Through an AI Harness — and the Biggest Finding Wasn't a Vulnerability

GURU Conference starts Nov 12: two days of email tactics, 100% free and virtual, powered by Constant Contact. Save your free spot.

Alright, let's get into it.

If you've been in this game for more than five minutes, you've had the same conversation I have. A client or a recruiter or somebody on Discord asks the same thing: "Can AI do pentesting now? Is my job gone?"

And the honest answer is — it depends what you think pentesting is.

If you think pentesting is firing off a nmap -sV and pasting the output into a template, then yeah, AI eats that for breakfast. That work was already dead, it just hadn't been buried yet.

But if you think pentesting is deciding what's actually true about a target, staying inside scope, and writing something a human being will pay money for — that part is alive and well. And AI turns out to be genuinely, uncomfortably good at it. Not because it replaces you. Because it lets one person run a workflow that used to need a small team.

So let me show you exactly what I did. Real engagement, real target, real report — and I'll show you the part where the whole thing nearly went sideways, because that's the bit everyone else leaves out of the video.

Here's the environment.

DeepSeek Harness (@deepseek-ai/dsh) is a harness — the layer that sits between a language model and a real machine. That distinction matters, so let me be clear about it, because people conflate these two things constantly.

A raw chat window can tell you to run dig. A harness can actually hold a PowerShell process, run the command, read the output, notice the result was unexpected, and decide what to do next. It has hands.

Code block

That boots the Web GUI as a local server and you drive the session in a browser. That's the mode I used for this whole engagement.

The reason I reached for a harness rather than a chatbot is the boring reason: I wanted a record. Every command, every result, and every decision in the session is a persisted transcript rather than something that scrolls away — and approval requests and their outcomes are logged too. On a client engagement, "the AI said so" is not a defence and it never will be — but "here is exactly what was executed, when, and against what" is something you can hand to a client, an insurer, or your lawyer.

Before the fun stuff, here's the honest inventory of what made this useful for security work — because not all of it is glamorous.

A real shell. It runs PowerShell on Windows and bash on Linux/macOS, and crucially, it runs each command inside a sandbox with modes you set deliberately. There's a locked-down workspace-write mode and a fully unrestricted mode. For client work you live in the restrictive mode and only escalate for the specific thing that needs it. More on why that's not just theatre in a minute.

Approval gates. Sensitive actions can be configured to require a one-shot human decision before they run. The ask policy pauses and asks. The never policy rejects outright. Every request and outcome lands in the session audit log.

Subagents. You can delegate independent chunks of work to child agents and collect structured results back. For pentest work this is the sleeper feature. Recon against five hosts in parallel? That's five subagents, not five sequential afternoons.

Workflows. When the delegation pattern gets big — audit forty files, check fifty subdomains — you don't orchestrate it by hand. You write a JavaScript orchestrator that fans work out across many subagents with phases and structured returns. This is the "I have a hundred of these and I want them done before lunch" tool.

Skills. Reusable, task-specific instruction sets loaded on demand. This is where your methodology lives. OWASP WSTG checklist, your CVSS scoring conventions, your report structure, your scope template — package it once as a skill and every future session starts already knowing how you work.

File presentation. When the deliverable is done, you declare it and the client opens the actual file. No copy-pasting out of a chat window, no markdown that renders like a ransom note in Word.

MCP. It speaks the Model Context Protocol, so you can bolt on outside tool servers — your own scanners, ticketing systems, whatever. This is the extensibility door and it's wide open.

None of that is a magic pentest button. It's a workshop. What you build in it is your problem — which is exactly how it should be.

Authorisation was in place, in writing, for a passive external vulnerability assessment of an Australian sporting association's public web presence. Two things were supplied: a domain, and an IP address.

That second one is where this story actually lives.

I set the rules of engagement in the session up front — target list, passive-only constraint, no authentication attempts, no fuzzing, and a hard rule that anything belonging to the hosting platform was out of scope.

Then I wrote down what "passive" actually meant for this job, because "passive" is one of those words that means five different things depending on who's saying it:

That list is the whole engagement. If it isn't written down, it isn't a constraint — it's a vibe.

Then I went to work.

Here's the thing I want juniors to tattoo on the inside of their eyelids: most of a passive assessment is reading things that were already public. You're not breaking in. You're paying attention.

First question — what is this domain, actually?

Code block

Straight away, the picture changed. The site wasn't sitting on infrastructure the client owned. It was on a hosted platform behind a CDN. Nameservers pointed at the host's DNS. That single observation reframes the entire report: this client does not administer the web tier. They rent it. Which means most of what a traditional pentest would test — server config, CMS patching, database hardening — isn't theirs to fix, isn't theirs to break, and absolutely shouldn't be reported as their vulnerability.

That's not a technical finding. That's a commercial finding, and it's the kind of thing that separates a tester from a scanner jockey. Your client is paying you to tell them what's true about their risk, not to hand them a list of things they cannot action.

Then I went after the boring goldmine — the security headers.

Code block

Which gave me the full response picture, and let me build a presence/absence checklist against the standard set: HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP, CORP.

Then the DNS hygiene checks, which on small-business and not-for-profit targets are always worth the two minutes:

Code block

NXDOMAIN. No DMARC record at all. No SPF either. Domain sends no mail, publishes no policy.

Stop and think about what that means for a club or a small business. Anyone on earth can send email that appears to come from that domain, to their members, their parents, their sponsors, and their bank, and there is nothing in DNS telling any receiving mail server to reject it.

No payload. No exploit. No CVE. Just a missing TXT record — and it's the most usable attack in the entire report, because it doesn't require touching the website at all.

Then certificates, straight off the TLS handshake:

Code block

This gives you the subject, the issuer, days-to-expiry, and the SAN list — which is quietly one of the best subdomain discovery techniques there is. A cert with a SAN you didn't know about is a host you didn't know about.

And the passive public-source checks that cost nothing:

Sitemaps in particular. On this target, the sitemap enumerated individual staff and board member profile pages by full name.

Counts only, no harvesting, and I'll tell you why in a second.

Where does a popcorn brand go after $20M in lifetime revenue? For Popsmith, it’s a projected $9M in 2026, up about 73% from 2025, and a nationwide launch in Williams Sonoma stores this holiday season. Own part of it before the raise closes.

Learn About the Raise

Now — the IP address.

I ran the same lookups against it that I'd run against any target. Reverse DNS, then registry attribution.

Code block

Reverse DNS came back as a static broadband address in a suburban Australian pool. Not hosting. Not a CDN. A residential or small-business static connection.

Then I attributed the block properly against the regional registry, and confirmed it: an ISP end-user pool, ALLOCATED NON-PORTABLE, no forward DNS record anywhere pointing to it.

I then checked the one thing that actually settles the argument — does the target's own DNS point at it?

Code block

The apex resolved to the hosting platform's own range. The www host resolved to a CDN address pool. Neither of them was anywhere near the IP I'd been handed — and no forward DNS record anywhere pointed at it either.

I also confirmed the address wasn't merely idle-but-theirs: a single TCP connect to it on 80 and 443 produced no response at all. Nothing was listening. Whatever that address is, it isn't a web server this client runs.

The IP address I'd been given had nothing to do with the client.

Now think about what the "efficient" version of this engagement looks like. You trust the scope document, you point a scanner at the supplied IP, and you let it rip.

You have just port-scanned a stranger's home internet connection.

In Australia that's not a stern email from a vendor. Under the Criminal Code Act 1995 (Cth), unauthorised access to, or impairment of, a computer is an offence — and "the client told me it was theirs" is not a defence when the client was wrong and you didn't check. Your authorisation covered the association. It never covered whoever lives at that address.

I reported it as the highest-rated finding in the document. Not because it's a technical vulnerability — there isn't one — but because it's the finding with the potential to end your career and your insurance cover. That's a real risk to the client's engagement, and it belongs in the report.

And here's the part I want you to actually take away. That finding came out of passive work. Registry lookups and DNS queries. No packets aimed at the wrong host, because I never sent any.

Which brings me back to the harness decision I mentioned earlier.

Two things about how I ran this session are the reason the story has a good ending.

One: the restrictive sandbox. By default, commands ran confined. That's not paranoia, that's exactly the shape of the mistake above. When your tooling makes you state what you're allowed to touch, you're forced to think about it. On a normal box you'd run a scanner and find out you were wrong afterwards.

Two: the approval gate. Sensitive actions can be configured to stop and ask a human before they proceed.

And I'll be straight with you — for this engagement I ran the session with approvals disabled, because it was passive-only and I'd defined the scope myself. That was defensible for this job.

I would not do that again on an active engagement. Let me say that more plainly, because it's the most important sentence in this post:

The reason the scope error was caught here is that I went looking for it deliberately, in the first ten minutes, before anything active happened. If I'd been running active tooling with approvals off, "I went looking" becomes "I already fired."

Approvals on for active work. Every time. If a client or a boss pressures you to turn them off because they're slowing the run down, that pressure is the finding.

Here's the thing nobody tells you when you're learning: the report is the deliverable, not the scan.

Your client cannot read a nmap output. They cannot action a CVE list. They're paying for a document that tells them what to fix first, who owns the fix, and what it means for their business. A tester who can't write is a tester who stays cheap.

So after the technical work, I had the harness go to work on the document itself.

The brief was specific — match an existing client report template exactly. Same green section banners, same dark finding headers, same fonts, same cover page, same header and footer treatment, same risk-classification table. Then populate it with my findings and do not invent anything.

The interesting part is how that got solved. Rather than trying to re-create the styling from a description — which produces an approximation, and a client can always tell — the session opened the reference document as a package, read its actual style definitions, theme colours, numbering definitions and headers, and reused them. Then it replaced only the body content.

The result is a genuinely native document, not a lookalike. Live chart with the severity counts, real headings, a table of contents that Word regenerates, page numbers that actually match.

It also verified the output rather than trusting it — schema-validating the generated file against the Office format and rendering pages to check them visually. That's the same discipline you'd apply to a manual report: don't hand a client something you haven't looked at.

Practical upshot for your own work: if you've got a house report template you deliver in, this is a one-time job to package it. After that, every engagement writes itself into your format.

Let me answer the question properly now.

What it did well: the grind. DNS work, header analysis, certificate parsing, public-source enumeration, cross-referencing, formatting findings, producing a document to a template. It did hours of that in minutes, and it did it cleanly.

Where it earned its keep: it followed instructions about staying passive. It didn't wander off and start "helpfully" probing things. When I said out of scope, things stayed out of scope. When the hosted platform's edge rate-limited me with a 429, it stopped and reported that rather than retrying in a loop — and "I stopped rather than hammering your provider's edge" is a line you can put in a report without flinching.

Where you are still, permanently, required:

And one more, because it matters on passive work specifically: discipline with personal data. That sitemap handed me a list of named volunteers. It would have been trivially easy to hoover up their contact details and pad the report. I counted entries and stopped. That's not squeamishness, it's professionalism — you do not need a volunteer's mobile number to prove the point, and once it's in a report it's in a report forever.

If you're building a consultancy — and if you're reading this on tsip.app you probably are — here's the part that should actually change your week.

A passive assessment is a product. It needs no intrusive testing, no scanning tools, no disruption window, and no platform provider's permission. It's fast, it's low-risk, and it consistently produces findings a client cares about and could not have found themselves. It's a perfect first paid engagement, a perfect retainer line item, and a perfect way to prove value before you ever quote them for an internal test.

Speed is a margin story. Two days of recon and reporting compressed into an afternoon isn't just convenience. It's the difference between a job you can't afford to take and one you can take three of. In this business, margin is time, and time is what a harness gives back to you.

Templates are the leverage. Your report format is your product surface. Package it once and every engagement inherits it. Clients don't buy tools, they buy the thing they get handed at the end.

Controls are your professional insurance. Sandboxes, approval gates, audit logs. Not because they're fashionable — because when something goes wrong, "here's exactly what was executed, on whose authority, and when" is the difference between a bad day and a legal problem.

Most GTM problems aren’t one big failure. They’re one or two weak spots quietly dragging down the rest.

The free GTM Audit scores your go-to-market across four dimensions, then sequences a 90-day roadmap for your stage.

Drop your URL. Get your score in about 10 minutes.

Get My GTM Score

Five things. Actually do them.

The tools changed. The job didn't.

AI didn't hand me a pentest. It gave me back the time I was burning on reconnaissance, formatting, and document wrangling — which is the time you should be spending on judgement. The scope error, the platform-versus-client call, the impact narrative, the decision to stop counting names: all of that was the work.

Ironically, the highest-rated finding in the whole engagement wasn't a vulnerability at all. It was somebody checking an assumption that was handed to them as fact — and finding out the client didn't actually know what they owned.

That's the job. Always was.

Now go check your own SPF record.

Before you run anything: This is written up from an authorised engagement with written permission and a defined scope. Every assessment needs that, in writing, before a single packet moves. Never test anything you don't have explicit authority over — including, and especially, an IP address somebody assured you was theirs.

If you want the workflow, the report template, or the checklist I use — it's in the TSIP downloads, and I go deeper on this stuff in the newsletter.