- Unsupervised Learning
- Posts
- The Future of Pentests, Bug Bounties, and Security Testing
The Future of Pentests, Bug Bounties, and Security Testing
People are very confused about the bounty vs. penetration test debate. They see fundamental differences that don’t actually exist, and they’re blind to what’s actually coming.
The reality of testing needs can be reduced to a few key variables, which exist on a spectrum.
Good vs. bad testers
Many testers vs. few testers
High vs. low business context
Paid by finding vs. by report
High tester trust vs. low tester trust
The future
Rather than giving my ideas about the future of testing, let me tell you what I believe to be the future of IT. Combined with the testing variables above the path should illuminate itself.
Businesses will eject most internal IT functions, preferring to use vendors instead.
The business will retain a small number of super elite IT people who are extremely fluent in both business and IT.
These business/IT (BIT) people will manage vendors in order to best achieve the goals of the business.
90% of IT workers will work for vendors / consultancies.
Infosec becomes vastly more data-driven in terms of what works for security and what does not, driven by insurance companies being the first groups incentivized to collect and use this information.
Insurance companies will determine the infosec standards because they will have the data about what works.
Because they will have the data, they know that certain projects need certain types of testing, and other types need other approaches.
Based on the type of project you have, the project’s business sensitivity, how many times its been assessed in the past, etc., there will be a best-fit type of assessment for that project.
The variables will be:
How sensitive the project is, i.e. the trust level of testers required to work on it.
Automated vs. manual testing.
How many testers are used.
Incentivization / payment structure.
The knowledge of the business required to provide valuable results.
The BIT person will reach out to several vendors and request an assessment with the precise mixture of these components.
Some vendors will excel at specific areas, such as high-trust testers, or testers who know a particular business, but the trend will be towards large companies that can do all of them.
Many large testing vendors will really be exchanges that can find any combination of individual to fit a given need.
The BIT will pick one vendor that has the best mix, and the work will get done.
Back to the present
So the future of testing is not a race to differentiation, it’s a race to similarity. Both penetration test companies and bounty companies need to become flexible enough to handle this entire range of capabilities.
Some assessments need highly vetted people, even if it’s just one or two. Other assessments need large numbers of people, no matter their background or alignment. Some require deep relationships with, and knowledge of, the customer. Others need no context whatsoever.
The truth is, as a BIT, you don’t care who you’re using as long as you can trust the results and that they’ll be professional. If you can provide better results, and better guidance on how to reduce risk for the company—all without breaking that precious trust that the whole thing is based on—you will do well.
Now, who do you think is better positioned to make this move?
Or, put another way, is it easier for:
Security services companies with deep relationships with companies built over years or decades to add a researcher program that brings hundreds or thousands of testers under their banner at varying levels of trust, and to then build/buy a platform for taking managing them finding bugs for their customers, or…
For companies based around a vulnerability platform to build the internal trust required to be trusted for ANY type of project the customer has?
I think it’s it’s the former. It would seem to be easier for a trusted security services company to add testers than for pure-play bounty companies to engage deeply into companies as a trusted advisor. But either way, that’s what the race looks like. And both company types must ultimately do both or face extinction.
Longer term It’s all about the testing talent
The funny part is that, long term, it actually doesn’t matter which model wins between pure-play bounty and traditional testing companies. The race described above is only on the 2-10 year scale. The next evolution of the future of work presents a threat to testing companies themselves—traditional, bounty, or whatever. Ask yourself this:
Who are the most important parties in the testing conversation?
The tester and the customer.
Everyone else is a middleman, i.e., a bunch of taxi companies in a world of ride sharing.
There is, of course, a component of, “Who are you going to sue if something goes wrong?”, and right now that dynamic heavily favors having a reputable testing company (not a bounty company) between the tester and the customer. But as the individual-based economy (and the technology-based trust infrastructure that powers it) gains acceptance, this will quickly decline as a factor.
As I talk about in The Real Internet of Things, individuals will be rated by trust, by quality, by how pleasurable they are to work with, etc., and they will win or lose contracts based on this rating.
As the infrastructure grows for tracking such meta, including one’s trustworthiness, how well they perform, how well they communicate, etc., having the middle-person will be needed less and less. The better the middle tech layer becomes at finding matches and ensuring quality, the less a third party is needed between the customer and the actual provider of the service.
Unsupervised Learning — Security, Tech, and AI in 10 minutes…
Get a weekly breakdown of what's happening in security and tech—and why it matters.
In short, both the traditional testing and bug bounty companies represent the old, taxi model of staffing security engagements, and they’re both going to be replaced by the individual-based gig economy.
That’s why I laugh when I see the industry so obsessed with the distinction between being penetration tester, a researcher, a bounty player, or whatever specific title we wish to assign. In an individual-based economy this distinction becomes arbitrary.
Testers will be testers with a set of skills. They might have a regular-ish job with a particular company, while they’re also doing other contracts on the side, while they’re also pursuing their own research as well. What are they? Pentesters? Bounty people? Researchers?
Yes, yes, and yes.
The future of security testing is individual-based and non-binary, and if you’re a third party in between them and the customer, you’re going to be in a bad position.
Summary
People are far too emotional about the bounty vs. pentest debate, usually because of bias.
The industry is actually racing to similarity, with all companies having many testers and many trust levels.
In the longer-term future it won’t even be about pentest or bounty companies because testers will be non-binary participants in the gig economy.
In this model, both types of companies become part of the past because they are third-party middlemen in a gig-based transaction.
This is why I can’t get too worked up about the bounty vs. pentest debate anymore. In the overall story arc of where security testing is going, it’s a moot point. Both models are intermediary, and the future is coming.
I look forward to the purity that individual-based testing will bring. It will simply be people with skills and reputations being harnessed to solve problems. And that’s the future of work, not just security testing.
Let’s stop fighting about who’s better at the old models, and start thinking about how to get to the new one.
Notes
Keep in mind that this transition will take time and will have many different phases. There will still be entities that pop up to pre-filter resources, like exchanges, that companies can buy from. But all of these solutions are temporary fixes to the technological problem of purchasers not being able to fully trust the rating systems. As those systems approach a realistic representation of quality and trust, the third-party vouching and liability services will become less needed and less valuable.
Insurance will be another solution to the liability problem. I can imagine a thriving insurance market where highly rated individuals run with insurance policies that help their clients relax about using them. So not only will they have high ratings in dependability, trustworthiness, and results quality, but they’ll also be covered for millions of dollars in the event of something bad happening. This will further diminish the need for a third party in the middle to take on liability.
I’ve had these thoughts for years now and have been reticent to share them. For one, I work at a testing company. Second, one of my favorite humans in the world (Jason Haddix) works at Bugcrowd, and my buddy Jeremiah Grossman is an advisor for them as well. Plus I have many other friends there that I care about, so I want to see all of them, as well as my own company, thrive. But there’s politics surrounding the topic—politics that get worse when marketing departments get involved and start slinging poo at each other. This happened recently, coming from the bug bounty companies, and I decided to write this as a reminder that the whole debate is an exercise in deck chair placement on the Titanic. Let’s be smarter and better.
If you’re wondering where this meta on individuals will be stored, such as their testing quality, their trustworthiness, their dependability, etc., I think the answer is in large, universal tech layers like LinkedIn, FICO, Insurance companies, etc. It’ll be all about massive databases of people, transactions, ratings, and fraud detection and defense. These companies will link job seekers with job providers, and everyone will run the WORK app on their phone like ride share drivers do now. Except it’ll be for all of your skills, not just one of them. This is how testers will find gigs—they’ll come to them automatically based on the customer’s need combined with their skillset, just like Uber and Lyft find drivers based on where you need to go, at what time of day, for how many people.
Many of these concepts are talked about in more depth in my book, The Real Internet of Things.
There will still be a place for companies that provide vetting services, but those services will not be consumed as sources for contractors, but rather authoritative tagging of resources with a seal of quality. So rather than saying, “Go get me N testers from X company.”, it’ll be, “Find me N testers who have the following criteria plus the X seal of quality given by Y service.”