I had a discussion recently with Johannes Ullrich regarding the definition of a penetration test. The conversation, as they often do, started out on standard VA vs. pentest ground, but ended up more of a focused debate of the definition of penetration testing itself.

The conversation started when he expressed his view that someone who does a pentest for a single goal–like harvesting a database or gaining domain admin–is not being thorough, and that they should continue on finding more and more flaws until as many as possible can be discovered (within reason).

That’s my summary of his position, but I urge you to read his position in his own words over at The Application Security Street Fighter Blog.

A Clear Distinction

So, my view is quite different. I feel the test type he’s describing is a detailed vulnerability assessment–not a penetration test. I think the key difference is opacity. He says over in his post that he doesn’t believe in black box testing, and that’s fine; I too agree that white box testing is far more effective.

But I think very concept of white box vs. black box is, by default, a discussion about vulnerability assessment and not penetration testing because penetration testing implies black box. In other words, you can have a white box or black box vulnerability assessment, but you can only have a black box pentest.

This is because the analog to the pentest (in my opinion) is the elite military unit commissioned to test a military base’s security. See Tiger Team. Many readers may remember Richard Marcinko, from SEAL Team 6 who used to break in and kidnap Admirals and hijack nuclear subs. This is a pentest in my view–you are given a single goal by the client: “get as far as you can”, or “see what you can get out of my database” or “try and modify my payroll records”.

The mission of the pentest team is to achieve the goal that has been given–not to find all (or even many) vulnerabilities in the target’s defenses. Any vulnerability assessment that takes place against the target will be solely for the purpose of finding a way in–nothing more. And if they get in on the first try, and accomplish the goal, then the report will indicate as much.

The report will state how they got in, what the customer should do to shore things up from that vector, and perhaps mention a few other things in passing. But it won’t be a comprehensive review of the customer’s security posture. That would be a separate engagement–a vulnerability assessment engagement.

Main Points

So here are my main propositions:

• A vulnerability assessment focuses on breadth: the goal is to identify as many issues as possible. This is why white box VAs are generally a superior option if the testing team is skilled enough to provide one.

• A penetration test focuses on depth, not breadth: the focus is on achieving a pre-determined goal that could only be possible if security were to fail, and not to find vulnerabilities. Vulnerabilities are used in a pentest, but they aren’t the focus. The focus is achieving the pre-determined goal.

• Vulnerability Assessments are (or should be) requisitioned by those who already know they have many issues and simply need help identifying and prioritizing them. The more issues identified the better, so naturally a white box approach should be embraced when possible.

• Penetration Tests should only be commissioned as a final stage of a security program, by those who believe their security program is mature, functioning well, and wish to test it against an adversary. It is, by nature, black box because the goal is to simulate a real-world attacker.

So, that’s my view of vulnerability assessments vs. penetration tests. As I told Johannes, I could be mistaken (this is semantics, after all), but only the opinions of our fellow professionals will be able to tell us which perspective on these two types of testing is more accepted.

Feedback welcome. ::

Only this one is more lamerer. This will take a list of words from a list you provide and append them to the front of a provided domain to see if they are valid hostnames. The idea is that you can then cat the output into a master list of things to scan:

hostfind.tar.bz2 hostfind.tar.bz2.sha1 hostfind.tar.bz2.sha1.asc

It took me nearly as long to package this thing as it did to write it. Unfortunately, the packaging is way more leet than the program itself — kind of lame considering there’s no README or anything…

I was working under the idea of, “if you make bzipped tarball of a 10 line shell script and sign it, it’s real software.” Turns out that’s not the case. I checked it again after I got done packaging it and it was still useless.

Anyway, I’m going to be incorporating this “module” into my bigger mst project, which is actually halfway decent in terms of being a time-saver (unlike this hideous token of boredom).

[The really sad part is that I use blog posts like this one as an archive system so that I can find this stuff later. I have plenty of places to put it where it won't get lost, but being able to search for it on my site is just too convenient.]

The other day I was in the middle of doing something very invasive at an organization during a penetration test and I was struck with a thought: “Why is this so easy?” The answer was immediately obvious:

It was easy because I knew I could go to the CSO if I got caught.

Were I to be there illegally, i.e. without permission from top management, I probably would have had a much harder time pulling off the acting. I think pentesters should keep this in mind when they get the urge to claim that social engineering is easy.

I’m starting to get more opportunities to use social engineering as part of penetration testing jobs, and after a recent success in the Southeastern United States I began pondering something:

Is it easier or harder to do social engineering in the South?

When you first think about it your gut reaction is that it’s easier, but it turns out that it’s all based on what type of attack is being performed. Getting information over the phone and such is most likely much easier, but attempting to physically access a building and roam around might actually be harder. Here’s why.

Southerners are very personable people. They want to know who’s working near them, who just got fired, who the new person is, etc. They don’t often work in close proximity to someone without having made contact with them in some way, shape, or form. This often manifests as extreme kindness, i.e. inviting new acquaintances to eat with their family, etc.

For a pentester trying to go unnoticed, this presents a problem. As I was on one of these engagements earlier this week I wondered if it would be easier in say, the Northeast, where, as I understand, people commonly don’t care at all who the people are around them.

But then I realized that while Southerners are more likely to be familiar with those around them, they’re also probably less likely to challenge someone who’s not supposed to be somewhere. I ran into this during this job as well; someone found me in their server room and didn’t say anything, most likely for fear of being rude.

Anyone have any additional anecdotal evidence to offer?