- Unsupervised Learning
- Posts
- Information Security Assessment Types
Information Security Assessment Types
There are many different types of security assessments within information security, and they’re not always easy to keep separately in our minds (especially for sales types).
For more primers like this, check out my tutorial series.
What follows is a brief description of the major types of security assessment, along with what differentiates them from commonly confused cousins.
Security assessment types
Vulnerability Assessment: A vulnerability assessment is a technical assessment designed to yield as many vulnerabilities as possible in an environment, along with severity and remediation priority information.
Commonly Confused With: The vulnerability assessment is most often confused (and/or conflated) with the Penetration Test. This is primarily because sales people think the latter sounds cooler, bless their hearts.
Best Used When: The vulnerability assessment is best used when security maturity is low to medium, when you need a prioritized list of everything that’s wrong, where the goal is to fix as many things as possible as efficiently as possible.
Penetration Test: A Penetration Test is a technical assessment designed to achieve a specific goal, e.g., to steal customer data, to gain domain administrator, or to modify sensitive salary information.
Commonly Confused With: The Penetration Test is most often confused (and/or conflated) with the vulnerability assessment. See ‘Sales People’ for more information. Another way to think about this is to imagine vulnerability assessments as looking for security problems when you know/assume they exist, and penetration testing as validating a configuration when you believe it to be secure.
Best Used When: Because a Penetration Test is designed to achieve one or more specific goals, they should not be commissioned by low or medium security organizations in most cases. Performing a Penetration Test against a low or medium security shop will simply yield recommendation all-time-greats like, “Implement patching across the organization.”, “Disable inactive users.”, and—one of my favorites—”Understand where your sensitive data is.” Do not waste money on a Penetration Test unless you’ve already undergone one or seventeen vulnerability assessments and then remediated everything that was found. Penetration Tests are for testing security that is assumed to be strong, not documenting the contents of a soup sandwich.
Red Team Assessment: A Red Team “assessment” is something of a misnomer in the corporate context since corporate Red Team services should ideally be continuous rather than point-in-time. So it should ideally be more of a service than an assessment. But regardless of that distinction, the central purpose of a corporate Red Team is to improve the quality of the corporate information security defenses, which, if one exists, would be the company’s Blue Team. In fact, that’s what a lowercase “red team” is: an independent group that challenges an organization to improve its effectiveness. In the case of corporate Red Teams, the org they’re improving is the Blue Team.
Red Team services should, in my opinion, always have the following five elements: Organizational Independece, Defensive Coordination, Continuous Operation, Adversary Emulation, and Efficacy Measurement.
Commonly Confused With: Red Team services are most commonly confused with Penetration Testing. Sales and marketing groups are using the terms nearly interchangeably, as are many internal security groups. People confusing the two are basically seeing “Red Teaming” as a sexier, more elite type of Penetration Test. They are not the same. A Penetration Test is a defined, scoped, and point-in-time assessment that has specific goals for success or failure. A corporate Red Team (whether internal or external) is a continuous service that emulates real-world attackers for the purpose of improving the Blue Team. They may share TTPs at times, but they have very different purposes.
Best Used When: Red Team services are best used when an organization has covered the basics of strong vulnerability management and has at least some capability to detect and respond to malicious or suspicious behavior in the environment. If an organization is still struggling with basic asset management, patching, egress traffic control, and other fundamentals, it’s usually best that they get those solved before hiring or building a “Red Team”. Red Teams are for testing mature security postures in a real-world way, not for enumerating issues in low-maturity environments. If you don’t have a Blue Team, you probably don’t need a Red Team.
Audit: An audit can be technical and/or documentation-based, and focuses on how an existing configuration compares to a desired standard. This is an important point. It doesn’t prove or validate security; it validates conformance with a given perspective on what security means. These two things should not be confused.
Commonly Confused With: Audits are often confused with pretty much any other type of security assessment where the goal is to find vulnerabilities and fix them. That could be part of an audit, if there’s an item in the standard that says you shouldn’t have vulnerabilities, but the key attribute is mapping current state against an arbitrary standard.
Best Used When: Organizations use audits to demonstrate compliance. Importantly, compliance should not be used to demonstrate security. Secure organizations are significantly more likely to be compliant (if checked), but compliant organizations should lay no claims to being secure just because they are in accordance with standard X or Y.
White/Grey/Black-box Assessment: The white/grey/black assessment parlance is used to indicate how much internal information a tester will get to know or use during a given technical assessment. The levels map light to internal transparency, so a white-box assessment is where the tester has full access to all internal information available, such as network diagrams, source code, etc. A grey-box assessment is the next level of opacity down from white, meaning that the tester has some information but not all. The amount varies. A black-box assessment—as you’re hopefully guessing—is an assessment where the tester has zero internal knowledge about the environment, i.e. it’s performed from the attacker perspective.
Commonly Confused With: The largest source of confusion around white/grey/black-box nomenclature is not realizing that they aren’t really an assessment type but rather an aspect of one. They’re most commonly paired with vulnerability assessments where you’re trying to find the most issues possible, and that provides significant incentive to open the curtains a bit. Remember that the goal of a vulnerability assessment is to find as many issues as possible, so hiding internal information from a tester that keeps them from finding issues doesn’t hurt them—it hurts you. Don’t confuse wanting to know what attackers can see/do with wanting to know what problems you have. These are two separate things and need to be approached separately. If you want to know what an attacker can do, fix all your issues until you’re confident you’re as secure as possible, and then get a Penetration Test.
Best Used When: White-box assessments are best used with vulnerability assessments because you want to find as many issues as possible, regardless of how the tester came to discover them. Grey-box assessments are often used when people are confused about the difference between a Penetration Test and a vulnerability assessment. They want to give some information, but not all. Let’s be clear: if you’re trying to find all of your issues, you shouldn’t withhold information from the tester. If you’re doing a Penetration Test, however, you shouldn’t give the tester anything, which is a black-box assessment. Keep these clear in your mind and you’ll be ok.
Risk Assessment: Risk Assessments, like threat models, are extremely broad in both how they’re understood and how they’re carried out. At the highest level, a risk assessment should involve determining what the current level of acceptable risk is, measuring the current risk level, and then determining what can be done to bring these two in line where there are mismatches. Risk Assessments commonly involve the rating of risks in two dimensions: probability, and impact, and both quantitative and qualitative models are used. In many ways, risk assessments and threat modeling are similar exercises, as the goal of each is to determine a course of action that will bring risk to an acceptable level.
Commonly Confused With: Risk Assessments are commonly confused with threat assessments, as both are pursuing similar goals. The primary differentiator is in where assessments start and where they place their focus. Threat Models focus on attack scenarios and then move into the agents, the vulns, the controls, and the potential impacts. Risk Assessments often start from the asset side, rating the value of the asset and the map onto it the potential threats, probabilities of loss, the impact of loss, etc.
Best Used When: Risk Assessments should arguably be considered an umbrella term for determining what you have of value, how it can be attacked, what you would lose if those attacks were successful, and what should be done to address the issues. It’s important that when someone says they’re going to do a risk assessment that you delve deeper into exactly what is meant by that, i.e. what approach or methodology will be used, what the artifacts will be, etc.
Threat Assessment: A threat assessment is a type of security review that’s somewhat different than the others mentioned. In general it pertains more to physical attacks than technology, but the lines are blurring. The primary focus of a threat assessment is to determine whether a threat (think bomb threat or violence threat) that was made, or that was detected some other way, is credible. The driver for the assessment is to determine how many resources—if any—should be spent on addressing the issue in question.
Commonly Confused With: The term “threat” is used numerous ways within security, which leads to significant confusion. In this case the term is used as in, “a threat was made”, or “determining whether the threat was real”, as opposed to the “threat-agent” usage. The origin comes from the Secret Service investigating school violence, and the challenge was determining which of the thousands of threats they received they should respond to with extremely limited resources. This is in stark contrast to what many think of when they hear threat assessment, which is investigating potential threat-agents, such as hackers, governments, etc.
Unsupervised Learning — Security, Tech, and AI in 10 minutes…
Get a weekly breakdown of what's happening in security and tech—and why it matters.
Best Used When: A threat assessment is best used in situations where someone has made a claim around performing an attack in the future, or such a potential is uncovered somehow. The goal in that case would be to learn whether the situation is worth spending resources on addressing.
Threat Modeling: Threat Modeling is not a well-understood type of security assessment to most organizations, and part of the problem is that it means many different things to many different people. At the most basic level, threat modeling is the process of capturing, documenting, and (often) visualizing how threat-agents, vulnerabilities, attacks, countermeasures, and impacts to the business are related for a given environment. As the name suggests, the focus often starts with the threat agent and a given attack scenario, but the subsequent workflow then captures what vulnerabilities may be taken advantage of, what exploits may be used, what countermeasures may exist to stop/diminish such an attack, and what business impact may result.
Commonly Confused With: Threat Modeling is confusing in general. Much of the confusion comes from debates around definitions and semantics, as threat modeling often includes discussions around threats, threat-agents, vulnerabilities, exploits, controls, risks, and impacts. Each of these is loaded on its own, and when you start trying to have a conversation with all of them at the same time religious wars often result. The other issue is that people lose track of the goal because there are so many elements in play. Are we trying to identify vulnerabilities? Are we trying to profile threat-agents? Are we documenting potential business impacts? Etc. The best way to summarize is to say that Threat Modeling brings a dose of potential reality to a security posture. It shows you, through attack scenarios, where gaps exist that could lead to real-world consequences.
Best Used When: Organizations should be using threat modeling early and often, and they should definitely be part of the development process. They are a way of ensuring that known potential attack scenarios can actually be handled by a given security posture. They can also be extraordinarily illuminating from a pure documentation and visibility standpoint. Seeing your potential threat-actors, how they’re likely to attack your app or system, using what vulns and what exploits, and what it’ll likely do to your organization is often a sobering experience. They’re especially useful for showing non-security-people how compliance and security products do not a security program make.
Bug Bounty: A Bug Bounty is a type of technical security assessment that leverages crowdsourcing to find vulnerabilities in a system. The central concept is simple: security testers, regardless of quality, have their own set of strengths, weaknesses, experiences, biases, and preferences, and these combine to yield different findings for the same system when tested by different people. In other words, you can give 100 experienced security testers the exact same testing methodology and they’re likely to find widely different vulnerabilities. The bug bounty concept is to embrace this difference instead of fighting it by harnessing multiple testers on a single assessment.
Commonly Confused With: Bug bounties are a relatively new approach to doing technical security testing, and there is some confusion around whether they should be done instead of another security test or in addition. The best answer, I’d argue, is that a bug bounty should be considered a vulnerability assessment in its goal of finding as many issues to remediate as possible, but be considered a Penetration Test in that you should do classical vulnerability assessments first. The reason for this is that bug bounties, because they use many people, excel in finding uncommon and eccentric issues, and the exercise is somewhat wasted on identifying the common problems that can be uncovered using automation and single-tester assessments.
Best Used When: Bug bounties are best used when you have already performed one or more standard vulnerability assessments (which should have included both automated and manual testing) and then you’ve remediated everything that was found. Consider them an optional step between classical vulnerability assessments and a Penetration Test, which, as noted above, does not seek to find all issues but rather to confirm that the security posture is where it needs to be by pursuing specific goals.
Most Frequently Confused
Here are some of the most common mistakes made when thinking about these assessment types.
If you aren’t confident in your security posture and know already that it’s not solid, you should be doing Vulnerability Assessments—not Penetration Testing. Penetration testing is for testing your posture once you have it where you want it.
The best way to think about Bug Bounties is an enhancement to the discovery phase of a Vulnerability Assessment. Vulnerability Assessments have two pieces: Discovery (finding as many issues as possible), and Prioritization (ranking what should be fixed first). Bug Bounties are great at the first part, and not good at the second. As such, they are best used when you have done multiple Vulnerability Assessments already and have already found the easy stuff. Bug Bounties excel at finding issues not found using other methods.
Because marketing and sales drive the infosec industry, people are constantly conflating Red Teaming and Penetration Testing. Because Red Teams are meant to emulate the adversary they generally only work if they are both continuous and run over long periods—ideally permanently. So if you have some company offering to do a 2-week “Red Team” engagement, this is probably better described as a Penetration Test. So the key distinctions are the emulation of real-world attackers, including their tenacity, the permanent duration of the attack, the TTP sophistication, etc. Assessments that lack those elements are Penetration Tests, not Red Team engagements.
Summary
Here’s the short version.
Vulnerability Assessments are designed to find as many vulnerabilities as possible for the purpose of prioritizing remediation efforts. The output is a list of prioritized issues.
Penetration Tests are designed to determine whether an attacker can achieve specific goals when facing your current security posture, such as stealing sensitive data or other activities that would harm the organization. The output is a report stating whether the goals were achieved or not, and any other observations that might have been made along the way. Penetration Tests do not provide a complete list of vulnerabilities or necessarily any prioritization of what was found; it’s mostly a yes or no for achieving the agreed-upon goals.
Red Teams are are designed to continuously and effectively emulate an organization’s real-world attackers for the purpose of improving its defensive capabilties. Red Teams operate continuously, with near-full-scope and very limited restrictions, and constantly evolve their approaches to match and/or exceed the capabilities of the organization’s actual attackers.
Audits are are designed to determine how a given organization measures against a given standard. Audits, as a rule, do not test security directly, but rather test compliance with a standard. The standard being tested against might have a strong or weak link to actual security, and should not be confused with a Vulnerability Assessment or Penetration Test. The output of an Audit is a list of areas that must be fixed in order to achieve compliance.
White/Grey/Black-box Assessments are a measure of how much information is being provided to a security testing organization during an assessment. These can be internal, external, application-based, network-based, with or without exploitation, etc. The only consideration for $SHADE-box assessments is the amount of information being shared with the testing party.
Risk Assessments are for determining the most important risks facing a given organization for the purposes of ensuring that they are brought within acceptable levels for the business. They can take many forms, but the output is always a list of prioritized risks followed by recommendations.
Threat Assessments are for determining whether a given threat (often, but not necessarily, physical in nature) is worth spending limited resources on. Output is usually a recommendation of what—if any—amount of effort should be dedicated to the issue.
Threat Models are for determining the various threats, threat scenarios, threat-actors, vulnerabilities, exploits, controls, and impacts that are related to a given system. They are ideally performed early and often during the creation process and can also be repeated after significant changes. Output often includes documentation of each of the above, along with residual risk after controls are considered, combined with recommendations for improvement.
Bug Bounties are projects that leverage crowdsourcing for the discovery of vulnerabilities in a system. They are a tool in the vulnerability assessment toolbox. The techniques used by those participating in a bounty can vary widely, as can the type of system being tested. The important part is that instead of an internal team, or a particular set of contracted employees doing the work, it’s instead a large collection of independent researchers who all bring their own perspectives to the testing.
Notes
Thanks to Saša Zdjelar and Jason Haddix for reading drafts of this article and providing feedback.
A number of assessment types aren’t listed here, such as web, mobile, code review, etc. They’re not listed because their names pretty clearly indicate what they are and what they’re used for.
Many people object to calling real testing a Vulnerability Assessment because of the history of vulnerability *scanning*, i.e., running Nessus and not doing any manual work. Because of this, Vulnerability Assessments have a bad name. We need to push back on that because it really is the most accurate way to describe a test where the goal is to find as many issues as possible and then rank them for remediation.
One common use for a Penetration Test, when a vulnerability assessment is actually needed, is to demonstrate that a problem actually exists. Word of advice: if management doesn’t listen to you when you tell them the 400 serious issues in the environment, and they need external validation, your next step should be LinkedIn. Lack of management support is one of the worst vulnerabilities you can have.
In the world of threats and vulnerabilities there has traditionally been a divide between threat-agents being the purview of government, and vulnerabilities being the purview of private industry. This is simply a matter of logistics. Companies don’t have the ability to track and prosecute attackers across state and country borders, and governments often do. So corporations have traditionally focused on the part of the problem that they can control.
Image by rkyr of Deviant Art