One of the best talks I saw at Enigma 2017 was this one by Ian Haken (@ianhaken) of Netflix about establishing a trust scaffolding within the cloud.

When building complex IT infrastructure you always have to create/retrieve/and store secrets, and when you’re doing this quickly, at scale, like we do in modern cloud environments, you have to start wondering where the trust chain begins.

Ian does a great job of explaining this in his talk, using the metaphor of turtles standing on turtles. Eventually you realize it’s turtles all the way down, and it gets a bit hopeless.

A solution

Ian explains that the solution to the problem comes from AWS itself.

In order to grant secrets to something or someone, you have to first confirm their identity. There are many ways to do this, but none of them really scale well in ephemerally paced cloud environments. But AWS has an API where an instance can ask who it is and what it’s meant to do.

AWS then returns a signed document attesting to the identify of that instance, and the instance can then send that document to the service that grants secrets, whether that’s some microservice, HSM, or whatever.

Summary

1. Secret management is hard.
2. Traditional ways of solving the problem tend to be impractical in ephemeral and large scale cloud environments.
3. The stack of turtles ultimately comes down to identity validation at the lowest level.
4. AWS is able to validate the identify of any instance in a programable and verifiable way.
5. You can use that capability to bootstrap the pulling of keys from a secret service.
6. BONUS: You can even bootstrap additional secret service instances using the same method.

I think this talk by Ian should be a lot more popular than it is. There are thousands of companies out there struggling with this same issue and that aren’t aware of a provider-based solution to the turtle problem.

If you know anyone grappling with secret management in the cloud, make sure they’re aware of this talk.

Notes

1. As Ian noted, the solution here is not specific to AWS; it can be any similar service that has the infrastructure to guarantee the identity and characteristics of a given instance.