Does Next-gen AppSec Require Next-gen Developers?

I just got back from AppSec Cali, which is quickly becoming my favorite infosec conference. The venue is fantastic. I know so many of the people, and they tend to be super humble and laid back. And the content is decent as well.

It’s just a great conference.

Anyway.

One of the big themes that was discussed this year (it’s been talked about in previous years, but it’s getting much stronger now) is the concept of developer enablement in top-performing organizations.

I’ve been tracking this trend across multiple industries for a couple of years now, and we see the trend heavily focused in places like Netflix and Facebook, i.e., the top-end of development shops where functionality is primary and the product is super respected.

These are a characteristics that tend not to exist in other types of organization.

• Developers are given massive amounts of responsibility and leeway.

• They tend to be able to push code to production pretty easily, often whenever they want.

• They do what works, and have very few forced boundaries in terms of languages, frameworks, specific rules to follow, etc.

• But they are ultimately responsible for the quality of what they produce, so if they produce insecure or otherwise feeble code, that’s on them.

• They’re responsible for whatever harm they cause.

• They can be fired easily.

That’s a fascinating combination of elements, and they also produce some interesting behaviors from security.

Security at the high-end

Security in an organization like this becomes the waterboy in a football game. They’re there to facilitate. Do you have everything you need? Is there anything I can get for you? What can I do for you to make your job easier?

Imagine if the waterboy walked up to the quarterback during a big game and was like:

The quarterback would look at this person like they had a head injury, and then they’d gesture slightly and that person would be banished from the stadium.

That’s an appsec group in a high-end development shop.

Security is not in charge. Their purpose is to enable the athletes to perform well. That means giving them the tools they need to be their best, and to be as safe as possible while doing so.

The athlete analogy

The athlete comparison continues to bear fruit, actually, because star athletes are gods until they’re kicked off the team.

Being able to push code to production whenever you want is a lot like having the freedom during a game to run in the wrong direction with the ball in order to get around a defender. It’s all fun and games until you get caught in the backfield.

You can do whatever you want, and you can make a couple of mistakes. But when you make one too many you’ll be tapped on the shoulder, and that’ll be it for you.

That’s true whether you’re showboating and dropping balls or you’re pushing crap code to production and causing outages during peak times.

The benefits

So why are elite companies heading in this direction? What’s so attractive about it?

Simple: companies with developers who are empowered in this way are able to produce better product.

As I’ve written about a number of times before, Evolution beats Design when creativity is the goal.

Old style organizations are Design Based. The “good” ideas come from above. From on high. From the mountain. The lowly engineers simply implement the plans of their betters.

And the result is often mediocrity.

The new model is bottom-up. Evolution style. With evolution the power comes from the bottom, from the people. And the developers are the people. They’re the artists. They’re the creators. They’re the producers.

When they are enabled they can produce more ideas which then mix powerfully with other ideas. Mutation occurs, tests are performed, and outcomes are created that blow away most anything produced by top-down teams.

That’s the benefit. That’s what Netflix, Facebook, and places like Riot Games have figured out. And they’re embracing it.

So why doesn’t everyone do it?

Now we arrive at the point of this piece.

I engage with many very large organizations in my consulting work—Global 50-100 companies often with thousands or tens of thousands of developers.

So the question is,

I think the answer is (mostly) no.

When Netflix, Facebook, and Riot Games do their hiring for developers (and their security team) they’re filtering for a special combination of tech and culture. You have to be in the top n percent in terms of tech skills, AND have this spectacular ability to take responsibility, exercise good judgement, be a team player, etc.

Most companies don’t have these kinds of standards. Not anywhere close. Much of these developer workforces are actually contractors. They’re giant swarms of low-paid resources that get dragged and dropped onto projects with very little vetting.

It’s not even the same sport as what the elite groups are doing in terms of hiring.

My worry

I’m somewhat concerned about the gap between mainstream, corporate development and the Holy Grail of Netflix/Riot Games (as it relates to developer productivity/responsibility/etc.).

It’s one thing to talk about DEVOPS and Agile and all this new high-speed Kung-fu, but it’s quite another to roll it out in organizations full of developers that simply can’t handle it.

I do believe that there are many in a pool of say 10,000 that CAN handle it. But we don’t know who those people are because we didn’t filter for those characteristics when they were hired. And it’s somewhat reckless to simply create projects, throw non-vetted folks onto the project after some training, and tell them to act like Netflix developers.

The more you move to new-style development (empowered/continuous/low-friction/etc), the more you move responsibility downwards towards the developer, and my feeling is that this shift is going to require a corresponding increase in developer quality.

Let’s play with some numbers.

Let’s say that top-tier orgs hire 1 of every 100 applicants that could get a job at other mainstream development companies, like Accenture or whatever. 1%. They discard 99% due to not being technical enough or not having the right mindset.

[ NOTE: I’ve no idea if that’s a reasonably accurate percentage or not, but I wouldn’t be surprised if it was even lower. ]

And let’s say you have a regular, corporate organization with 1,000 developers, and they’ve been told to “Move to a Netflix model.”, or to otherwise get to some nebulous approximation of the elite dev shops we’ve been talking about.

What happens when you give average developers maximum autonomy? What happens when you give them all the superpowers of ultimate responsibility, and the keys to production?

Now obviously this isn’t some switch you’re going to throw, and suddenly give a bunch of people access they’re not used to. You’re not going to do this overnight.

But my point is that we’re likely to have to go through a similar vetting process as these other companies used, to find the developers—out of that 1,000—who are capable of handling the new model.

And we’re most likely going to have to filter using an evolutionary model rather than a design one. You’re going to have to try people out, in other words, and be willing to discard them if they don’t work out. That’s likely going to be a big switch for traditional companies.

We probably won’t have to get to the 1% filter level to make major progress. Maybe we can take the top 10%. Or maybe the top 25%. I don’t know where that bar is, but it’s definitely not going to be the entire pool, and it will very likely be way less than half.

Summary

1. The new model for high-output and high-quality development is based on empowering developers with massive amounts of creative control, power, and responsibility.

2. This requires a different type and quality of developer.

3. The cutting-edge companies in the new model spend massive amounts of effort finding these people from within already qualified talent pools.

4. Because developers of this quality are so hard to find, it’s going to be far harder than people think to move the software industry over to the new style of product development.

TL:DR: The more creative freedom and responsibility you give to developers, the higher quality they need to be, and there are only a small percentage of the overall developer pool that will make that cut. For this reason, we might want to temper our expectations for some mass migration to Netflix-style software development in the corporate world.

Notes

1. When I say that Netflix or Facebook or Riot Games does this or that, I’m speaking fairly generally in a way I think is safe based on what I know from various contacts I have in these companies. But I’m not claiming to be some authoritative resource on their exact hiring criteria. If you have insider information that runs counter to any of this, do let me know.

2. I am also a strong believer in the concept that giving responsibility to people makes them smarter and better in MANY ways, so this effect will definitely help convert many of the lower-quality developer resources into rockstar artist types. In truth those people always were that; they just never had a chance to show it. This will help with the move, but even then I think we’ll have massive percentages of developers who simply can’t make the transition.

3. I’m a security guy and could very well be wrong about some of these difficulty levels. Happy to have my models corrected by someone who knows better.

Related posts:

1. My Journey to Beginner Audiophile

2. The Future of Pentests, Bug Bounties, and Security Testing

3. The Relative Importance of High-Resolution Audio, CD Quality, and MQA