Infosec: Vulnerability Assessment vs. Penetration Test

I had a discussion recently with Johannes Ullrich regarding the definition of a penetration test. The conversation, as they often do, started out on standard VA vs. pentest ground, but ended up more of a focused debate of the definition of penetration testing itself.

The conversation started when he expressed his view that someone who does a pentest for a single goal–like harvesting a database or gaining domain admin–is not being thorough, and that they should continue on finding more and more flaws until as many as possible can be discovered (within reason).

That’s my summary of his position, but I urge you to read his position in his own words over at The Application Security Street Fighter Blog.

A Clear Distinction

So, my view is quite different. I feel the test type he’s describing is a detailed vulnerability assessment–not a penetration test. I think the key difference is opacity. He says over in his post that he doesn’t believe in black box testing, and that’s fine; I too agree that white box testing is far more effective.

But I think very concept of white box vs. black box is, by default, a discussion about vulnerability assessment and not penetration testing because penetration testing implies black box. In other words, you can have a white box or black box vulnerability assessment, but you can only have a black box pentest.

This is because the analog to the pentest (in my opinion) is the elite military unit commissioned to test a military base’s security. See Tiger Team. Many readers may remember Richard Marcinko, from SEAL Team 6 who used to break in and kidnap Admirals and hijack nuclear subs. This is a pentest in my view–you are given a single goal by the client: “get as far as you can”, or “see what you can get out of my database” or “try and modify my payroll records”.

The mission of the pentest team is to achieve the goal that has been given–not to find all (or even many) vulnerabilities in the target’s defenses. Any vulnerability assessment that takes place against the target will be solely for the purpose of finding a way in–nothing more. And if they get in on the first try, and accomplish the goal, then the report will indicate as much.

The report will state how they got in, what the customer should do to shore things up from that vector, and perhaps mention a few other things in passing. But it won’t be a comprehensive review of the customer’s security posture. That would be a separate engagement–a vulnerability assessment engagement.

Main Points

So here are my main propositions:

• A vulnerability assessment focuses on breadth: the goal is to identify as many issues as possible. This is why white box VAs are generally a superior option if the testing team is skilled enough to provide one.

• A penetration test focuses on depth, not breadth: the focus is on achieving a pre-determined goal that could only be possible if security were to fail, and not to find vulnerabilities. Vulnerabilities are used in a pentest, but they aren’t the focus. The focus is achieving the pre-determined goal.

• Vulnerability Assessments are (or should be) requisitioned by those who already know they have many issues and simply need help identifying and prioritizing them. The more issues identified the better, so naturally a white box approach should be embraced when possible.

• Penetration Tests should only be commissioned as a final stage of a security program, by those who believe their security program is mature, functioning well, and wish to test it against an adversary. It is, by nature, black box because the goal is to simulate a real-world attacker.

So, that’s my view of vulnerability assessments vs. penetration tests. As I told Johannes, I could be mistaken (this is semantics, after all), but only the opinions of our fellow professionals will be able to tell us which perspective on these two types of testing is more accepted.

Feedback welcome. ::