- Unsupervised Learning
- Posts
- Four Thoughts on Making Good Metrics
Four Thoughts on Making Good Metrics
I was just asked to give some insight on metrics and came up with these four observations. I thought I’d share them here.
The key thing to realize is that metrics have to be connected to two things in order to have value—on one side they’re connected to your goals, and on the other side they’re connected to actions. If you’re tracking metrics for things that aren’t tied to what you’re trying to achieve, you’re failing. And if you’re tracking metrics for things that you have no intention of ever acting on, you’re also failing. The primary questions should be, “why am I tracking this, and what do I intend to do when I get a certain result?”
One of the most important information security metrics, and one that’s commonly overlooked, is the percentage of X that is under management. Many programs spend significant time detailing the risk they know about, but don’t attempt to capture the amount of visibility or coverage they have overall. This can lead to a significantly false sense of security and the ignoring of massive, fertile attack surface within the environment.
A demand for perfect accuracy in risk measurement is often counterproductive. Remember that the most important part of the metrics equation is the action that results from them, not the metrics themselves. If a metric is near enough to accurate to allow for a proper adjustment in behavior, it’s accurate enough to be a metric.
Don’t confuse your metrics with a big data project. One may feed the other, but they should usually be quite different things. There is a tendency to over-granularize metrics simply because it might be possible and the data could be useful at some point in the future. The biggest difference is that what you track should have roughly the same number of separations that you have in your possible behavior changes. If you can only do one of two things as a response to a metric, don’t spend time and energy getting the metric broken down into 16 shades. Two will do nicely.
If you have something you’d add to this list, or if you disagree with any of them, reach out and let me know.